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Abstract 


This  thesis  describes  the  Instructional  System  Development 
(ISD)  process  as  it  relates  to  missile  combat  crew  member  (MCCM) 
training  at  the  Strategic  Air  Command  units.  Written  due  to  the 
lack  of  ISD  literature  applicable  to  operational  units,  it  is 
intended  to  serve  as  a  practical  guide  for  instructors  and  staff 
members  at  the  missile  unit,  enabling  them  to  understand,  apply, 
and  control  ISD  in  the  development  of  MCCM  job  training  with 
greater  confidence  and  efficiency.  Its  description  is  based  upon 
careful  synthesis  of  related  literature,  telephone  interviews, 
and  a  comprehensive  survey  of  missile  unit  instructor  cadre. 

Specific  focus  within  the  thesis  is  on  the  ISD  models 
applied  at  the  missile  unit,  and  the  problems/benefits  of 
applying  ISD  to  MCCM  job  training.  Research  findings  indicate 
that  missile  unit  instructors  modify  the  specific  procedures 
recommended  by  two  ISD  models:  the  Five-Step  Model  and  the 
Criterion  Referenced  Instruction  Model.  The  problems/benefits  of 
applying  ISD  at  the  missile  unit  are  found  to  be  similar  to  those 
at  Air  Training  Command  units.  Overall  indications  are  that 
missile  unit  instructors  regard  ISD  models  as  useful  tools  for 
developing  and  conducting  MCCM  job  training. 


INSTRUCTIONAL  SYSTEM  DEVELOPMENT 
AT  OPERATIONAL  MISSILE  UNITS 

I .  Introduction 

General  Focus 

Instructional  System  Development  (ISD)  is  a  "systematic 
approach  to  training"  (12:10).  It  is  often  applied  when 
developing,  conducting,  or  evaluating  occupational  training 
programs.  Numerous  ISD  models  exist:  the  Interservice 
Procedures  for  Instructional  System  Development,  the  Four  Factor 
Approach  to  Job  Analysis,  the  Five-Step  Model,  and  the  Criterion 
Referenced  Instruction  Model  are  four  examples  of  such  models.. 

The  models  differ  in  specific  procedural  recommendations,  but 
have  similar  characteristics.  Each  model  is  characterized  by  its 
"deliberate  and  orderly,  but  flexible"  nature,  and  its  focus  on 
student  needs  (7:Al-3). 

Developers  and  conductors  of  training  programs  (referred  to 
as  instructors)  proceed  in  a  "deliberate  and  orderly"  fashion 
when  using  ISD  models.  Most  models  prescribe  the  ISD  process  in 
terms  of  a  sequence  of  well-defined  steps.  Using  the  steps 
prescribed  by  an  ISD  model,  instructors  systematically  "identify 
the  critical  job  task  that  must  be  learned,  the  place  where  the 
task  can  best  be  learned  (in  the  school,  in  the  unit,  or  on  the 
job),  and  the  method  of  instruction"  (20:2). 


The  steps  prescribed  by  each  ISD  model,  however,  are 
intended  as  "tools"  or  guidelines,  not  as  "restrictions"  (21:2). 
ISD  models  identify  a  constant  interaction  between  the 
environment  and  the  development  of  a  training  program,  and 
between  the  steps  of  the  ISD  process  itself.  This  interaction 
necessitates  occasional  modification  of  specific  ISD  steps  by  the 
instructor.  Furthermore,  ISD  models  encourage  instructors  to 
obtain  and  adopt  feedback  throughout  the  process  of  developing 
and  conducting  training  programs.  Such  flexibility  allows  the 
instructor  to  respond  to  the  environment,  the  internal 
constraints  of  the  specific  training  program,  and  to  the  needs  of 
the  students. 

ISD,  in  fact,  is  centered  on  the  needs  of  the  student. 
Students  are  generally  trained  for  a  particular  purpose. 

Depending  on  the  purpose,  certain  outcomes  are  required  of 
students  in  training  programs;  they  must  attain  specific 
knowledge,  be  able  to  perform  specific  tasks,  or  demonstrate 
specif ic  skills.  ISD  models  thus  demand  that  instructors 
identify  these  required  behaviors  (19:199),  then  promote  active 
student  involvement  in  developing  and  accomplishing  the 
behaviors.  Keys  to  active  student  involvement  include  the 
following:  allowing  individual  students  to  progress  at  their  own 
pace;  providing  ample  opportunity  for  students  to  practice  the 
necessary  behaviors;  and  ensuring  that  students  demonstrate 
accomplishment  of  the  behaviors  under  the  exact  conditions  and 
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standards  required  to  perform  adequately  outside  of  the  training 
environment.  By  promoting  active  student  involvement,  ISD 
fulfills  the  most  direct  need  of  students  -  assuring  that 
graduates  can  perform  the  behaviors  which  are  required  in  the 
real  world. 

The  above  characteristics  of  ISD  are  apparently  appealing  to 
the  military  services.  For  example.  Air  Force  Regulation  (AFR) 
50-8,  Policy  and  Gui dance  for  Instructional  System  Development , 
directs  that  the  ISD  process  be  used  for  the  development  and 
operation  of  all  its  training  programs  (2:1).  The  other  military 
services  have  likewise  developed  policy  requiring  the  application 
of  ISD. 

Problem  Definition 

Numerous  sources  of  literature  have  been  written  describing 
practical  applications  of  the  ISD  process  as  related  to  military 
training  programs.  Most  of  these  sources,  however,  portray  its 
application  at  units  where  the  primary  mission  is  training. 

Fewer  sources  describe  the  practical  relationship  of  ISD  to 
operational  units,  where  training  is  a  secondary  mission. 
Operational  units  thus  rely  solely  on  theoretical  or  general 
information  about  ISD  from  sources  such  as  Air  Force  Manual  ( AFM ) 
50-2,  Instructional  System  Development,  and  Air  Force  Pamphlet 
(AFP)  50-58,  Handbook  for  Designers  of  Instructional  Systems. 
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This  lack  of  practical  literature  for  operational  units  is  an 
unfortunate  circumstance  for  instructors  at  such  units. 

Relegation  of  training  to  a  secondary  mission  at  operational 
units  means  that  fewer  resources  can  be  devoted  to  training 
programs  than  at  training  units.  With  fewer  resources  to  devote, 
instructors  at  operational  units  could  benefit  from  more 
practical  literature  describing  ISD  for  their  particular 
situation.  Such  literature  could  ultimately  improve  the 
effectiveness  of  ISD  and  the  training  programs  at  operations  . 
units.  This  research  project  will  therefore  examine  and  describe 
the  ISD  process  as  it  is  related  to  one  type  of  operational  unit. 

Specific  Focus 

Goal.  This  research  seeks  to  develop  a  detailed 
description  of  ISD  as  it  relates  to  developing  and  conducting  job 
training  for  missile  combat  crew  members  (MCCMs)  at  operational 
Strategic  Air  Command  (SAC)  missile  units.  Such  a  description 
could  serve  as  a  practical  tool  for  instructors  and  missile  staff 
members,  enabling  them  to  understand,  apply,  and  control  ISD  in 
the  development  of  job  training  with  greater  confidence  and 
efficiency. 

Job  Training  Defined.  For  the  purpose  of  this 


research  report,  job  training  is  the  process  by  which  MCCMs 
develop  and  maintain  the  knowledge,  skills,  attitudes,  or 
behaviors  required  to  achieve  the  mission  of  the  Single 


Integrated  Operational  Plan  (SIOP),  which  is  to  maintain  nuclear 
equipped  missiles  on  constant  alert  in  preparation  to  launch  upon 
Presidential  orders.  Three  broad  categories  of  job  training  are 
examined  in  this  research  project:  weapon  system  training. 
Emergency  War  Order  (EWO)  training,  and  codes  training. 

Through  weapon  system  training,  MCCMs  develop  and  maintain  their 
skills  in  operating  the  Minuteman  or  Titan  weapon  systems.  Such 
training  enables  MCCMs  to  correctly  operate  equipment,  perform 
launch  procedures,  respond  to  hazards,  and  understand  nuclear 
surety.  EWO  training  provides  MCCMs  with  knowledge  of  the  SIOP, 
and  enables  them  to  identify  and  respond  to  valid  orders  which 
execute  their  sorties.  Codes  training  enables  MCCMs  to  operate 
the  specialized  launch  control  system  at  Minuteman  units.  Each 
of  these  are  further  categorized  into  more  specific  types  of 
training,  including:  recurring,  Missiles  Procedures  Trainer 
(MPT),  supplemental,  remedial,  unit  orientation  or  upgrade,  and 
transition  training.  Discussion  of  each  specific  type  will  be 
done  in  Chapter  II. 

Research  Questions.  Two  questions  foster  research  in 
pursuit  of  a  description  of  ISD  as  it  relates  to  MCCM  job 
training : 

1.  Is  any  particular  ISD  model,  combination  of  models, 
or  component  parts  used  at  the  missile  unit? 

2.  What  are  the  perceived  problems/benefits  of  using 
ISD  to  develop  and  conduct  job  training? 
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Though  these  specific  questions  do  not  foster  a  means  for 
investigating  every  conceivable  circumstance,  obtaining  answers 
to  them  will  provide  sufficient  information  to  analyze  ISD  as  it 
relates  to  developing  and  conducting  job  training  for  MCCMs  at 
operational  SAC  missile  units. 


■»  V  V 


II.  Background 


Overview 


This  chapter  summarizes  the  available  sources  of  information 
on  ISD  as  they  relate  to  job  training  for  MCCMs  at  operational 
SAC  missile  units.  It  begins  with  a  brief  identification  of  the 
types  of  sources  used  as  background  information  in  this  research. 
A  large  portion  of  the  chapter  is  then  devoted  to  describing  two 
ISD  models  accepted  within  SAC.  Next,  the  historical  basis  of 
ISD  acceptance  at  SAC  missile  units  is  discussed.  Following  this 
discussion,  missile  unit  training  personnel,  programs,  and 
environments  are  identified.  A  summary  of  thoughts  on  the 
usefulness  of  ISD  is  also  presented,  prior  to  the  conclusion  of 
this  chapter. 

Sources  of  Background  Information 


A  variety  of  sources  have  yielded  useful  background 
material.  General  descriptions  of  ISD  models  have  been  obtained 
from  books,  AFMs ,  and  courses.  Telephone  interviews  with  ISD 
experts  in  the  missile  career  field  have  been  used  to  obtain 
historical  information.  Information  regarding  the  specific 
personnel  and  programs  involved  in  the  missile  unit  training 


process  has  been  extracted  from  SAC  regulations.  Research 


studies  concerning  military  applications  of  ISD  were  also  useful 
sources  of  background  material  throughout  this  chapter. 


Perhaps  the  most  valuable  source  of  information,  however, 
has  been  this  author's  personal  experience  in  applying  ISD  at  the 
missile  unit.  This  author,  a  former  instructor  and  staff  member 
at  the  381st  Strategic  Missile  Wing  (SMW),  McConnell  Air  Force 
Base  ( AFB ) ,  Kansas,  has  six  years  of  experience  in  developing, 
conducting,  and  supervising  job  training.  This  experience 
includes  time  both  as  a  weapon  system  and  EWO  instructor,  and  as 
a  Wing  Instructional  System  Manager,  responsible  for  the  overall 
supervision  and  coordination  of  ISD  as  it  relates  to  job  training 
at  the  unit.  To  supplement  this  experience,  the  author  has 
received  ISD-related  training  including  the  "Development  and 
Management  of  Instructional  Systems"  course  at  Sheppard  AFB, 
Texas,  and  the  "Criterion  Referenced  Instruction  Training"  course 
at  Vandenberg  AFB,  California. 

ISP-Related  Methodologies 

ISD  Models.  Two  ISD  models  are  accepted  by  SAC  missile 
units.  One  model  is  the  USAF  Five-Step  Model,  as  prescribed 


CONSTRAINTS - 

LEGEND:  CURRICULUM  LOOP  f~"  ~  - 

FEEDBACK  AND  INTERACTION  LOOP 


Figure  1.  Five-Step  Model  (7:1-1) 

Five-Step  Model.  Perhaps  the  most  widely  taught  and 
used  model  within  the  AF  is  the  Five-Step  Model.  This  model,  as 
depicted  in  Figure  1,  is  designed  especially  for  the  technical 
and  job  training  programs  used  in  the  USAF.  The  process  depicted 
in  Figure  1  operates  under  environmental  constraints  for  the 
particular  training  system.  Such  constraints  include  time, 
money,  facilities,  equipment,  manpower,  and  the  unit's  mission. 
Instructors  must  recognize  these  constraints,  overcoming  them 
where  possible  and  working  around  them  when  necessary,  yet 
realizing  that  flexibility  is  permissible  when  using  the  Five- 
Step  model. 

Step  1,  ANALYZE  SYSTEM  REQUIREMENTS,  is  the  procedure  by 
which  human  performance  requirements  are  analyzed  and  documented 
by  instructors  when  developing  job  training  programs  (8:1-3). 
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When  performing  task  analysis,  instructors  may  directly  interact 
with  workers,  experts,  or  management  by  surveying,  questioning, 
or  observing  them.  Instructors  may  also  analyze  written  material 
about  the  job,  such  as  regulations,  checklists,  or  technical 
data.  Through  such  analysis,  the  instructor  compiles  a  list  of 
the  knowledge,  skills,  behaviors,  "duties  and  tasks,"  and 
"equipment  and  materials"  required  of  workers  to  perform  their 
jobs  in  a  satisfactory  manner  (1:74).  This  list,  known  as  the 
job  performance  requirements  (JPR),  forms  "the  base  on  which  to 
design  instruction  and  the  frame  of  reference  for  evaluating 
instruction"  (7:2-2). 

At  Step  2,  DEFINE  EDUCATION/TRAINING  REQUIREMENTS,  "the 

difference  between  the  JPRs  and  the  overall  abilities"  of 

potential  students  are  compared  (1:7-4).  Any  behavior  listed  in 

the  JPR  which  potential  students  cannot  already  accomplish  should 

be  considered  for  training.  The  goal  of  this  step  is  to  select 

tasks  for  training,  depending  on  numerous  other  considerations 

about  the  behaviors,  including: 

consequence  of  inadequate  performance;  task  del^y 
tolerance  (how  fast  the  person  must  react  to  status); 
frequency  of  performance,  complexity;  difficulty  of 
learning;  and  time  between  training  and  performance 
(1:7-4). 

Further  considerations  such  as  "availability  of  instructors, 
equipment  and  facilities,  long  lead  time  resources,  and  time  for 
training"  are  also  part  of  identifying  specific  tasks  for 
training  (1:7-4,  7-5).  When  the  specific  tasks  have  been 
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identified  for  training,  they  are  compiled  into  a  set  of 


instructional  standards.  As  described  by  AFM  50-2, 

Instructional  System  Development , 

These  instructional  standards  are  specialized 
publications  used  to  standardize  and  control 
instruction.  They  are  also  the  means  for  coordination 
between  those  who  are  responsible  for  instruction  and 
the  organization  for  whom  instruction  is  conducted. 

Such  coordination  helps  to  ensure  that  the  instruction 
is  designed  to  meet  the  using  organization's  needs. 
Instructional  standards  list,  by  means  of  task  and 
knowledge  statements,  (is)  the  instruction  needed  to 
qualify  personnel  to  do  the  job.  These  listings  are 
the  quantitative  requirements  to  be  met  by  the 
instructional  system.  Instructional  standards  are  also 
qualitative.  They  show  the  level  of  proficiency  to  be 
developed  for  each  task  knowledge  (7:3-7). 

Instructional  standards  are  the  culmination  of  Step  2  of  the 

model. 

Step  3,  DEVELOP  OBJECTIVES  AND  TESTS,  begins  the  process  of 
quantifying  the  instructional  standards  into  actual  programs  of 
instruction.  At  Step  3,  instructors  develop  objectives  for  each 
of  the  instructional  standards  of  the  previous  step.  Objectives 
are  specific  statements  labeling  each  behavior  required  of  a 
student,  the  conditions  under  which  each  behavior  must  be 
accomplished,  and  the  level  of  excellence  or  standards  for 
accomplishment.  These  statements  serve  important  purposes  for 
both  instructors  and  students.  For  students,  objectives  identify 
exactly  what  is  required  for  successful  accomplishment  of  a 
training  program.  The  student  can  gear  all  study  and  practice 
toward  accomplishment  of  the  objective  instead  of  non-essential 
behavior.  For  the  instructor,  one  purpose  of  objectives  is  to 
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"serve  as  a  basis  for  criterion  tests"  (7:4-3).  Criterion  tests, 
developed  at  this  step  of  the  model,  serve  as  the  basis  for 
measuring  the  student's  accomplishment  of  objectives.  They  often 
use  the  same  wording  as  the  original  objectives.  Criterion  tests 
are  not  always  written  measures,  however.  A  specific  objective 
may  require  a  student  co  physically  demonstrate  a  behavior;  for 
example,  demonstrate  ability  to  troubleshoot  or  fix  a  particular 
piece  of  equipment.  The  criterion  test  must  therefore  adapt 
itself  to  measure  each  objective,  as  specified  by  the  objective. 
Other  purposes  of  objectives  for  the  instructor  are  to  "serve  as 
a  basis  for  instructional  technology  decisions;  determine 
appropriate  content  of  the  instruction  (and)  serve  as  a  basis  for 
instructional  decisions"  (7:4-3).  These  decisions  are  actually 
made  by  the  instructor  during  Step  4. 

Step  4  of  the  model  is  titled  PLAN,  DEVELOP,  AND  VALIDATE 
INSTRUCTION.  There  are  three  important  aspects  of  planning 
instruction.  One  aspect  is  sequencing;  objectives  should  be 
arranged  "in  the  most  effective  sequence"  which  "produces 
required  learning  in  the  shortest  time"  (1:7-6).  Common 
acceptable  sequencing  alternatives  include  arranging  objectives 
in  a  simple  to  complex  order,  or  in  the  order  that  the  job 
requires  (7:5-3).  A  second  aspect  of  planning  involves 
examination  and  decision  by  the  instructor  about  methods  by  which 
students  can  progress  toward  achieving  objectives  (9:87).  All  ISD 
models  encourage  selection  of  student  self -pacing  and  student 


participation  methods  over  "lock  step"  methods,  where  all 
students  progress  at  the  same  rate  (7:5-5).  The  third  aspect  of 
planning  instruction  involves  selecting  media.  Media  are 
"channels  of  communication"  which  provide  the  student  with 
information  for  achieving  objectives  (9:87).  The  teacher  is  one 
particular  medium  through  which  students  receive  information; 
other  examples  include  textbooks,  programmed  text,  simulators, 
slides,  videotapes,  and  computers.  All  of  these  "present 
material  to  the  student,  and  when  successful,  make  it  clear  and 
interesting  so  that  the  student  learns"  (19:29).  Learning  is 
further  enhanced  by  selection  of  the  medium  or  combination  of 
media  which  not  only  give  information  about  the  objective,  but 
also  allow  for  student  practice  and  evaluation  in  achieving  the 
objective  prior  to  the  criterion  test  (7:5-5). 

After  planning  the  media,  methods,  and  sequence  of 
objectives,  the  instructor  develops  each  in  the  form  to  be  used 
for  instruction.  Other  final  courseware  such  as  lesson  plans, 
study  guides,  and  Plans  of  Instruction  are  produced  at  Step  4  of 
the  model.  When  final  courseware  has  been  developed.  Step  4 
recommends  a  tryout  training  program  on  a  sample  of  potential 
students.  This  tryout  compares  "student  performance  against  the 
performance  criteria"  of  the  objectives  (7:5-14);  in  other  words 
it  validates  the  training  program  (1:7-6).  Such  validation  is 
recommended  prior  to  formal  implementation  of  the  training 


program. 


As  training  is  formally  implemented,  the  model  moves  into 
its  final  stage.  Step  5  -  CONDUCT  AND  EVALUATE  INSTRUCTION.  At 
this  stage,  the  formal  training  program  is  conducted  as  a 
"student-centered,  individualized  instructional  system"  where 
instructors  act  as 

Managers  of  learning  and  fill  the  roles  of  classroom 
administrator,  tutor,  and  counselor.  They  are  now  able 
to  give  much  of  their  attention  to  the  needs  of  the 
individual  student  (7:6-1). 

Operation  of  the  training,  though  based  upon  thorough  analysis 
and  a  series  of  orderly  decisions,  is  not  allowed  to  stagnate. 

The  training  system  must  be  constantly  evaluated  for 
deficiencies.  The  five-step  model  evaluates  training  programs 
both  internally  and  externally  as  training  is  conducted.  The 
purpose  of  internal  evaluation  is  similar  to  validation  -  it 
allows  the  evaluator  to  judge  whether  the  training  system  still 
enables  the  student  to  achieve  the  objectives.  Internal 
evaluation  thus  monitors  the  performance  of  the  students  trained 
by  the  system  (12:89).  Additionally,  "evaluation  has  an  external 
function"  which  measures  "not  the  attainment  of  objectives  ... 
but  the  appr opriateness  of  the  objectives  themselves." 

External  evaluation  thus  examines  whether  the  tasks  required  of 
the  job  for  which  the  training  system  exists  are  still  the  same 
as  when  the  original  task  analysis  was  performed  (12:92).  Both 
external  and  internal  evaluation,  combined  with  the  conduct  of 
training,  provide  a  link  from  Step  5  to  previous  steps  of  the 
model.  It  implies  flexibility  in  the  model. 
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Flexibility  is  an  important  aspect  of  any  ISD  model.  The 
five-step  model  demonstrates  this  flexibility  in  a  number  of 
ways.  As  shown  by  Figure  1,  the  five-step  model  is  a  never- 
ending  process  where  the  end  links  to  the  beginning.  By 
demonstrating  a  never-ending  process,  the  model  accounts  for  a 
changing  environment  and  provides  a  continued  orderly  fashion  for 
redeveloping  training  based  upon  those  changes.  The  model  also 
demonstrates  flexibility  by  explicitly  linking  all  steps  of  the 
process  through  feedback  and  interaction  (again  seen  in  Figure 
1).  Feedback  obtained  at  any  step  of  the  process  enables  the 
instructor  to  identify  deficiencies  or  areas  of  enhancement  for 
the  training  program.  In  accordance  with  the  model,  the 
instructor  then  uses  the  feedback  information  to  identify  and 
accomplish  the  steps  which  will  improve  the  training  program. 
Often,  for  example,  instructors  find  inadequate  objectives  or 
test  questions  at  Step  4  when  they  are  validating  instruction, 
thus  requiring  a  brief  return  to  Step  3.  Finally,  the  model 
demonstrates  its  flexibility  by  requiring  instructors  to  document 
their  actions  as  they  accomplish  the  process.  Such  documentation 
enables  instructors  to  keep  track  of  the  interaction  between  the 
various  steps;  instructors  can  relate  actions  to  previous  actions 
and  to  objectives  in  developing  a  training  system.  The 
documentation  also  enables  the  various  persons  and  organizational 
entities  involved  to  coordinate  and  consolidate  their  efforts. 

As  will  be  demonstrated  throughout  this  research  project,  each  of 


these  flexibility  aspects  are  important  not  only  to  ISD  theory, 
but  also  to  practical  implementation  of  ISD  within  operational 
missile  units.  Missile  units  have  even  extended  the  flexibility 
of  the  five-step  model  by  applying  it  in  conjunction  with  another 
model.  Criterion  Referenced  Instruction  (CRI). 

CRI  Model.  The  CRI  model  of  Dr.  Mager  contributes  to 
ISD  theory  in  three  ways:  first,  it  adds  goal  analysis  to  ISD 
theory;  second,  it  expands  many  of  the  procedures  prescribed  by 
the  five-step  model;  and  third,  it  presents  information  in  a 
manner  consistent  with  ISD  theory. 

One  contribution  of  CRI  to  ISD  theory  is  goal  analysis. 

Goal  analysis  is  related  to  task  analysis  since  both  identify 
areas  of  potential  training.  Unlike  task  analysis,  however,  goal 
analysis  identifies  cognitive  areas  of  potential  training,  such 
as  attitudes,  rather  than  tasks  or  skills  required  by  persons. 

It  could  be  desired,  for  example,  that  individuals  show  "pride," 
"finesse,"  or  "enthusiasm"  in  their  work.  Once  such  goals  have 
been  identified,  it  becomes  the  task  of  the  instructor  to 
quantify  them.  Goals  must  be  subdivided  into  specific  student 
performances  which,  if  accomplished,  indicate  achievement  of  the 
goals.  The  CRI  model  recommends  that  the  instructor  ask  the 
following  question  when  quantifying  goals:  "If  these 
performances  were  accomplished,  would  I  be  willing  to  say  the 
goal  has  been  achieved?"  After  identifying  and  quantifying  goals 


through  goal  analysis,  the  remaining  procedures  are  similar  to 
the  five-step  model  (15). 

CRI  has  been  judged  by  at  least  one  set  of  authors  "to 
represent  more  of  an  expansion  and  clarification  of  specific  ISD 
tasks  and  subtasks"  than  a  departure  from  the  five-step  model 
(17:14).  This  "expansion  and  clarification"  is  a  second 
contribution  of  CRI.  Though  the  five-step  procedure  is  not 
depicted,  the  same  ideas  are.  For  this  reason,  a  detailed 
description  of  CRI  procedures  in  this  thesis  would  merely  be 
repetitive. 

The  final  contribution  of  the  CRI  model  to  ISD  theory  is  its 
manner  of  presentation.  ISD  theory  is  presented  in  CRI  courses 
and  books  in  the  same  manner  it  recommends  students  or  readers  to 
develop  their  training  programs;  CRI,  in  other  words,  practices 
what  it  preaches.  CRI  books  and  courses  begin  with  a  set  of 
objectives  for  the  audience  to  achieve  while  they  are  in  the 
process  of  learning  about  CRI.  Organization  of  the  CRI 
presentation  from  then  on  allows  the  audience  to  work  towards 
achieving  the  objectives.  First,  necessary  information  about 
each  specific  objective  is  presented  in  the  form  of  explanations, 
examples,  or  pictures.  Other  sources  of  information  are  usually 
also  identified  to  the  student.  Next,  the  audience  is  given 
practice  in  achieving  each  objective?  they  are  questioned  about 
each  objective  as  they  practice,  enabling  each  of  them  to  self¬ 
measure  progress.  Correct  answers  indicate  that  a  particular 
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member  of  the  audience  is  capable  of  achieving  the  objective.  0 
the  other  hand,  incorrect  answers  necessitate  additional 
information  and  practice,  which  the  CRI  book  or  course  provides. 
As  audience  members  complete  practice  at  their  own  pace,  each  is 
presented  with  a  criterion  test  as  a  final  measure  of  ability  in 
achieving  the  objective.  By  enabling  the  audience  to  apply  ISD 
theory  as  it  is  presented  to  them,  the  CRI  model  appeals  to  the 
audience's  emotions;  if  the  procedures  work  when  the  audience  is 
learning  about  the  topic,  the  audience  will  be  inclined  to  use 
the  procedures  when  developing  and  conducting  their  own  training 
programs. 


Historical  Basis  of  ISD  Acceptance  at  SAC  Missile  Units. 


Historically,  both  ISD  and  the  training  programs  of  the 
current  SAC  missile  systems  -  Minuteman  and  Titan  -  originated 
during  the  1960s.  The  timeframe  of  origin,  however,  was  the  only 
intersection  between  SAC  missile  unit  training  programs  and  ISD 
for  a  number  of  years.  This  trend  of  concurrent,  but  parallel 
development  continued  for  a  decade-and-a-half . 

ISD  originated  mainly  from  the  works  of  Dr.  B.P.  Skinner  in 
the  early  19  60s.  His  advocacy  of  "systematic  approach  to 
training"  and  courses  based  upon  instructional  objectives  were 
appealing  to  the  commercial  airlines.  American  Airlrnes  thus 
adopted  his  theories  in  the  19  60s  and  applied  them  successfully 
to  pilot  training.  This  successful  application  of  ISD  theory 
brought  similar  appeal  to  the  USAF.  In  1969,  AFM  50-2, 
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Instructional  System  Development#  was  published#  with  its 
five-step  model  based  upon  Skinner's  works  (13).  By  1980,  the 
following  USAF  policies  were  dictated: 

1.  HQ  USAF  will  promote  the  use  of  ISD  in  all  MAJCOMs. 

2.  ISD  will  be  applied  to  all  new  training  programs. 

3.  ISD  will  be  selectively  applied  to  existing  training 
and  education  systems  when  economically  feasible  (12:8). 

It  still  was  several  years  before  missile  units  implemented  ISD. 
From  19  70  to  19  77#  missile  unit  training  was  characterized  as 
lacking  sound  analysis  of  realistic  conditions  and  standards  for 
MCCMs  (though  a  JPR  did  exist#  as  did  a  general  "Proficiency  Code 
Key").  Instead,  "a  fascination  with  the  theoretically  possible 
overshadowed  realistic  training  and  evaluation  objectives" 

(14:4).  Attempts  were  made  to  train  MCCMs  to  higher  skill  levels 
than  realistically  required.  An  explanation  for  this  was  fear  of 
the  unknown  by  the  missile  units;  fear  of  the  unknown  tasks, 
conditions,  and  standards  that  higher  headquarters  inspection 
agencies,  such  as  the  SAC  Inspector  General  (IG)  and  the  3901st 
Strategic  Missile  Evaluation  Squadron  (SMES),  would  impose  upon 
MCCMs  when  evaluating  units  (14:6). 

Another  higher  headquarters  inspection  agency,  however, 
began  a  reversal  of  the  momentum  of  this  circums tance.  In  1977, 
the  USAF  IG  criticized  SAC  for  not  applying  ISD  at  the 
operational  unit  training  programs  (16).  This  criticism,  by 
itself,  was  an  important  factor  in  SAC’s  acceptance  and  use  of 


ISD .  The  criticism  foreshadowed,  however,  and  provided  a 
potential  solution  for  overcoming  the  decline  in  crew  member 
proficiency  in  SAC  between  1977  and  1979.  When  it  was  determined 
in  1979  that  "ineffective  training  programs"  were  among  the 
causes  of  this  decline,  the  ISD  process,  which  was  already  in  its 
initial  stages  in  SAC,  began  its  forward  movement  in  earnest 
(14: 7) . 

Initialization  of  the  ISD  process  in  SAC  began  with  a 
re-analysis  of  the  tasks  applicable  to  Minuteman  MCCMs  by  the  HQ 
SAC  training  agency,  the  4315th  Combat  Crew  Training  Squadron 
(CCTS).  During  this  re-analysis  in  19  78,  the  4315th  CCTS 
subdivided  each  task  into  a  list  of  short  phrases  which 
identified  the  functions  or  activities  making  up  each  task. 

After  this  list  was  provided  to  Minuteman  units,  it  became  the 
first  set  of  performance  objectives  in  SAC  which  could  be  used  by 
both  instructors  and  students  to  identify  what  was  required  of 
the  task  (16). 

Though  this  set  of  performance  objectives  was  useful,  it  was 
incomplete  at  the  same  time.  It  failed  to  prescribe  standards 
for  accuracy  or  timeliness  of  performance.  To  overcome  this 
deficiency,  a  working  group  consisting  of  3901st  SMES  and  4315th 
CCTS  personnel  was  formed.  In  1978  and  1979,  this  group 
developed  a  new  Proficiency  Code  Key,  identifying  four  skill 
levels  for  the  MCCM: 

"Level  1"  or  "Entry  Level"  where  the  MCCM  requires 


demonstration  and  assistance  to  perform  the  task. 

"Level  2"  where  the  MCCM  requires  assistance  only  on 
the  more  difficult  activities  of  the  task. 

"Level  3"  where  the  MCCM  can  accomplish  the  entire 
task  without  error  or  assistance. 

"Level  4"  where  the  MCCM  performs  the  task  "in 
established  time  parameters"  without  error  or 
assistance  (14:16). 

This  Proficiency  Code  Key  was  designed  so  that  a  specific  skill 

level  requirement  could  be  applied  to  each  specific  task 

designated  in  performance  objectives  (16). 

In  19  79,  the  working  group’s  emphasis  shifted  towards 

developing  a  set  of  Comprehensive  Performance  Objectives  (CPOs) 

for  the  JPRs  of  each  weapon  system.  Through  a  series  Of 

conferences  at  Vandenberg  APB,  California,  the  working  group  and 

representatives  of  each  SAC  missile  unit  performed 

a  complete  task/subtask  analysis.  This  included 
extensive  research  into  Technical  Order  narrative 
and  checklists,  regulations  that  direct  appropriate 
actions,  and  other  documents  that  govern  task 
performance  (14:17). 

The  CPOs  which  resulted  from  this  analysis  were  a  more  complete 
product  than  the  original  Minuteman  performance  objectives.  The 
CPOs  now  specified  for  each  weapon  system  a  realistic  set  of 
tasks  and  subtasks  required  of  crew  members,  the  conditions  under 
which  tasks  would  be  performed,  and  the  standards  of  performance 
based  upon  the  Proficiency  Code  Key  (16).  A  sample  CPO  is 
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depicted  m  Table  2.1,  and  mechanics  of  the  CPOs,  as  described  in 
an  Air  Command  and  Staff  College  report  by  Major  James  D. 
Luethemeier,  are  shown  in  Table  2.2.  Experts  within  the  command 
regarded  the  CPOs  as  the  result  of  Steps  1  and  2  of  the  Five-Step 
Model  (22).  Drafts  of  the  CPOs  were  sent  to  the  units  in  the 
fall  of  1980,  and  were  officially  implemented  in  the  form  of  SAC 
regulations  by  the  end  of  the  year  (16). 

Since  the  official  implementation  of  the  CPOs  in  1980,  job 
training  at  SAC  missile  units  has  been  based  upon  realistic  tasks 
with  better  defined  standards,  rather  than  the  old  "fascination 
with  the  theoretically  possible."  This  is  not  say,  however,  that 
the  CPOs  have  been  without  criticism  or  faults.  Their  refinement 
on  several  occasions  since  1980  attests  to  their  lack  of 
infallibility  as  a  missile  training  tool.  The  fact  is,  however, 
that  since  1980  the  CPOs  have  become  the  basic  tool  for  ISD 
application  at  SAC  missile  units. 

Missile  Unit  Training  Personnel ,  Programs,  and  Their 


Environments. 

Personnel  and  Programs.  The  persons  responsible  for 
developing  and  conducting  such  training  are  assigned  to  the 
Operations  Deputate  of  the  missile  unit.  Three  sets  of  persons 


are : 


Directorate  of  Training  (DOTI)  personnel:  tasked  with 


weapon  system  training. 


TABLE  2.1 


Sample  CPO  for  a  Titan  Unit  (6:3-159) 


006  Perform  Security  Alarm  Situation  Checklist 

.EVEL  « 4 « « 

PERFORMANCE 

CONDITIONS 

STANDARDS 

1.  Recognize  requirement  to 
accomplish  task 

Given  T.0.-1,  SACR  207-17.  SACCEM  2-2S,  local 
directives,  and  under  one  or  more  of  the  follow¬ 
ing  conditions: 

COBH 

4  4  4  4 

(within  3  minutes  of  status 
presentation) 

1.  Alarm  indications 

2.  Non-alarm  indications 
a.  I AW  T.0.-1 ,  fig  5-1 

b.  1AW  SACR  207-17 

4  4  4  4 

(Within  5  minutes  of  status 
presentation) 

2.  Oirect/process  checklist 
actions 

a.  Through  step  2 

4  4  2  2 

(As  stated  In  performance 
f3) 

b.  After  step  2 

3  3  2  2 

3.  Configure  equipment  and 
verify  indications 

1 .  CCBV 

a.  Closes  remotely 

4  4  2  2 

(within  1  minute  after  C/l 
Initiation) 

b.  Closes  manually 

2  2  4  4 

(within  5  minutes  a*ter  C/L 
initiation) 

4.  Notify  agencies/personnel 

3  3  2  2 

S.  Reconfigure  equipment  end 
exit  C/l 


1.  A.R.T.  investigation  terminated 


3  3  3 


TABLE  2.2 


The  Mechanics  of  the  CPOs  (14:17,  19) 


Le vel  Designator .  The  level  designator  above  the 
STANDARDS  column  identifies  the  highest  level  at  which  this  task 
will  be  performed.  It  relates  a  task  back  to  the  Proficiency 
Code  Key.  For  example,  a  Level  4  task  may  have  the  first  subtask 
performed  at  Level  4  with  all  other  following  subtasks  performed 
at  Level  3.  Normally,  only  one  overall  level  number  will  appear 
for  a  particular  crew  position,  however,  a  dual  level  appears  if 
the  JPR  could  be  performed  exclusively  at  one  level  or  another 
depending  on  conditions  presented.  A  Level  3/4  thus  indicates 
that  the  highest  level  attained  could  be  either  3  or  4  depending 
upon  which  listed  conditions  are  presented. 

Gi ven  Statement .  The  "Given  ..."  statement  at  the  top 
of  the  CONDITIONS  column  specifies  the  nonvariable  factors 
impacting  upon  the  entire  Comprehensive  Performance  Objective. 

It  includes  all  associated  materials  such  as  technical  data, 
regulations,  other  governing  documents,  and  equipment  hardware. 
Specific  permanent  hardware  such  as  the  MPT,  LCC,  equipment 
racks,  and  materials  normally  located  where  the  crew  member 
performs  duty,  are  assumed  to  be  present  and  functional  unless 
otherwise  indicated.  The  statement  further  includes  possible 
methods  of  entry  into  the  task  and  possible  restrictions  on 
problem  presentation. 

Performance  Column .  The  PERFORMANCE  column  identifies 
all  actions/decisions  the  crew  member  would  be  expected  to  take 
i"  performing  the  task.  Performance  listings  can  include  such 
actions  as  recognizing  the  need  to  perform  the  task,  configuring 
and  reconfiguring  equipment,  monitoring  the  results  to  determine 
normal  versus  abnormal  indications,  and  reporting  results. 

Conditions  Co 1 umn .  The  CONDITIONS  column  lists  the 
conditions  under  which  a  crew  member  could  be  expected  to  perform 
each  subtask.  Only  those  conditions  listed  are  considered 
appropriate  for  the  performance  being  accomplished.  The  absence 
of  any  condition  implies  that  the  performance  will  always  be 
required  when  accomplishing  the  task.  With  the  listing  of 
performances  and  conditions,  the  crew  member,  instructor  or 
evaluator  would  be  able  to  see,  it  was  hoped,  exactly  what 
performance  is  expected  under  listed  conditions  for  every  JPR. 

Standards  Co  1 umn .  The  STANDARDS  column  states  the  level 
at  which  each  subtask  must  be  performed.  It  is  determined  by  the 
Proficiency  Code  Key.  Any  special  standards,  such  as  Level  4 
time  constraints,  are  spelled  out. 


EWO  Training  Branch  (DQ22T)  personnel :  teach  SIOP 
procedures  to  MCCMs. 

Codes  Training  Division  (D09 )  personnel :  implement 
codes  training. 

Each  set  of  persons  differs  in  broad  training  responsibilities, 
yet  produces  the  same  specific  types  of  training:  recurring, 

MPT,  supplemental,  remedial,  unit  orientation  or  upgrade,  and 
transition.  Recurring  training,  conducted  on  a  monthly  basis, 
generally  consists  of  self-study  packages  for  all  MCCMs  to 
accomplish  at  launch  facilities,  followed  by  classroom  sessions  • 
in  a  training  facility  at  the  base.  Similar  to  recurring 
training  with  respect  to  its  regularity  for  all  MCCMs  (though  on 
a  quarterly  basis  rather  than  monthly)  is  training  at  the  MPT. 
MCCMs  receive  realistic,  hands-on  experience  for  procedures  which 
cannot  be  performed  regularly  at  the  launch  facility,  yet  are 
essential  to  mission  accomplishment.  This  training  includes,  for 
instance,  missile  launch  or  hazard  procedures. 

Supplemental  training,  the  third  particular  type,  is 
presented  on  a  non-recurring  basis.  This  training  is  provided  to 
all  MCCMs  when  a  change  occurs  that  has  an  important  immediate 
impact  on  weapon  system  or  procedures.  It  also  is  often  used  to 
correct  a  significant  deficiency  in  training  or  when  numerous 
MCCMs  have  been  identified  as  using  incorrect  procedures. 

Fourth,  remedial  training  is  also  presented  on  a  non-recurring 
basis  to  individual  MCCMs  with  the  purpose  of  correcting  a 
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deficiency  in  knowledge  or  procedures.  Unit  orientation  training 
(Minuteman  units)  or  upgrade  training  (Titan  units)  is  the  fifth 
particular  type,  and  is  used  to  familiarize  and  qualify  new  MCCMs 
on  the  unique  mission,  capabilities,  and  peculiarities  of  the 
unit  upon  their  arrival  after  a  generalized  introduction  to  SAC 
MCCM  procedures  at  the  4315th  CCTS.  Finally,  transition  training 
enables  selected  MCCMs  to  develop  skills  necessary  to  command  a 
missile  combat  crew. 

Other  persons  within  the  Operations  Deputate  impact  upon 

this  training.  Since  MCCMs  are  training  recipients, 

ISD-developed  training  must  be  centered  upon  their  job  needs. 

Standardization/Evaluation  Division  (DOV)  personnel  are  tasked 

with  evaluating  MCCMs  at  the  missile  complex  and  the  MPT.  In 

doing  so,  DOV  personnel  provide  feedback  on  the  quality  of  job 

training.  DOV  additionally  participates  in  the  review  and 

coordination  process  prior  to  approval  of  key  training  programs. 

Another  key  person,  the  Chief,  Instructional  Systems  Management 

Division  (ISMD)  (formerly  called  the  Wing  Instructional  System 

Manager),  coordinates  and  guides  all  matters  of  training  and 

evaluation  in  accordance  with  the  ISD  process.  Though  ISMD  is 

typically  a  one  or  two  person  division,  it  has  broad 

responsibilities  within  the  deputate: 

Help  ensure  effective  instructor  and  evaluator 
training  programs. 

Advise  instructors  on  lesson  plan  preparation  and 
test  construction. 

Advise  evaluators  on  evaluation  script  preparation. 

Review  all  instruction  and  evaluation  material  to 
ensure  program  quality  and  effectiveness. 
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Periodically  monitor  classroom  and  MPT  instruction 
and  evaluation  periods  to  ensure  program  objectives 
are  met. 

Coordinate  on  all  DOT  and  DOV  training  and 
evaluation  scripts  to  ensure  compliance  with  CPOs. 

Serve  as  a  focal  point  for  all  4315th  CCTS  formal 
training  ...  (10:3). 

In  discharging  these  responsibilities,  ISMD  educates  instructors, 
evaluators,  and  staff  members  on  "the  benefits  gained  from 
following  the  ISD  model"  (10:1).  The  Chief,  ISMD,  also  serves  as 
an  advisor  to  the  Deputy  Commander  for  Operations  (DO)  in 
training  and  evaluation  matters.  Such  advice  enables  the  DO  to 
provide  overall  policy  and  leadership  within  the  Deputate. 

Environment .  Agencies  outside  the  Operations  Deputate 
also  impact  MCCM  job  training.  At  the  missile  unit,  these 
agencies  include  the  Wing  Commander  and  his  staff,  the  Unit 
Command  Post,  the  Maintenance  Deputate,  the  communications  and 
civil  engineering  squadrons,  the  Security  Police,  and  the  Wing 
Safety  Office.  Each  of  these  agencies  interacts  with  MCCMs  in 
daily  operations.  Policy,  procedures,  or  equipment  changes 
affecting  such  agencies  often  also  affect  MCCMs  and  require 
training.  Additionally,  the  daily  interaction  enables  these 
agencies  to  provide  important  feedback  on  MCCM  capabilities  and 
deficiencies . 

Outside  the  missile  unit  are  higher  headquarters  agencies 
whose  policies  and  regulations  impact  MCCM  job  training.  Head¬ 
quarters  USAF  establishes  policy  regarding  ISD  in  AFR  50-8, 

Policy  &  Guidance  for  Instructional  System  Development . 


As  a  reminder,  it  was  criticism  provided  by  the  Headquarters  USAF 
inspection  agency,  the  IG,  which  began  the  ISD  wheels  turning  in 
SAC  in  1977  (16).  SAC,  as  the  major  command  responsible  for 
missile  units,  provides  the  greatest  impact  on  MCCM  training.  Its 
impact  is  discussed  below,  subsequent  to  identification  of  two 
other  higher  headquarters  agencies.  Numbered  Air  Force  and  Air 
Division.  Generally,  both  of  these  agencies  provide  staff 
assistance  to  missile  units  in  implementing  the  training  policies 
of  the  AF  and  SAC. 

Headquarters  SAC  impacts  every  aspect  of  MCCM  job  training. 
Its  training  agency,  the  4315th  CCTS  at  Vandenberg  AFB , 
California,  provides  a  continuous  pipeline  of  new  MCCMs  to  the 
missile  units  through  Initial  Qualification  Training.  Manned  by 
experienced  former  missile  unit  instructors  and  evaluators, 

4315th  CCTS  also  offers  four  formal  courses  on  developing, 
conducting,  and  evaluating  training,  including:  the  SAC  Missile 
Instructor  Course,  Criterion  Referenced  Instruction  Training 
(CRIT),  the  Mini-CRIT  course,  and  the  Wing  Instructional  Systems 
Manager  Training  Course.  A  description  of  each  course  can  be 
found  in  Appendix  A.  Each  course  is  available  at  specified  times 
upon  request  of  the  missile  unit.  The  only  persons,  however, 
required  by  regulation  to  attend  any  of  these  courses  are  ISMD 
personnel.  SACR  55-66,  Volume  I,  I CBM  Operations ,  requires 
them  to  attend  the  Wing  Instructional  System  Manager  Training 
course. 
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Several  SAC  regulations  impact  the  daily  training  activities 
of  the  missile  unit.  One  regulation  governs  each  broad  category 
of  training: 

-  SACR  50-16/  Volume  1/  I  CBM  Training ,  governs  waapon 
system  training. 

SACR  50-21/  ICBM  EWO  Training  Programs,  governs 
EWO  training. 

SACR  55-56,  Volume  I,  Control  of  Minuteman  Code 
Components ,  governs  Codes  training. 

Furthermore,  there  are  regulations  specifying  the  duties  and 
tasks  of  two  other  major  actors  within  the  missile  unit: 

SACR  55-66,  Volume  I,  ICBM  Operations ,  defines 
duties  of  ISMD  personnel. 

-  SACR  55-66,  Volume  II,  ICBM  Combat  Crew  Standardi zation 
and  Evaluation,  prescribes  detailed  procedures 

for  Stand/Eval  personnel  in  conducting  evaluations. 
Finally,  SACR  50-16,  Volumes  IV  through  XIII,  are  the  CPOs  which 
set  standards  for  MCCMs,  and  thus  are  the  basis  for  training  and 
evaluation. 

Headquarters  SAC  inspection  agencies,  the  3901st  SMES  and 
the  SAC  IG,  enforce  these  regulations  and  assess  the  quality  of 
MCCM  job  training  in  conjunction  with  regular  inspections  of  the 
missile  units.  A  specific  item  of  evaluation  for  each  agency  is 
the  unit's  compliance  with  SAC  regulations;  MCCM  job  training  is 
evaluated  for  compliance  with  the  regulations  listed  above.  Both 
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inspection  agencies  also  evaluate  MCCMs  at  the  MPT  and  missile 
complex  as  a  measure  of  the  technical  quality  of  MCCM  training. 
The  SAC  IG  additionally  evaluates  above  and  beyond  regulation 
compliance  or  technical  quality;  it  evaluates  the  management 
practices  of  the  actors  involved  in  MCCM  job  training.  Results 
from  both  IG  and  SMES  inspections  are  recorded,  then  forwarded  to 
all  SAC  missile  units.  SAC  thus  maintains  an  environment  of 
continuous  feedback  to  its  missile  units  in  an  effort  to  improve 
the  quality  and  usefulness  of  MCCM  job  training. 

Usefulness  of  ISP 

Benefits  and  problems  which  surround  the  application  of  ISD 
are  valuable  measures  of  its  usefulness.  Information  of  this 
nature  on  missile  units  is  scarce;  however,  Major  Luethemeier, 
the  author  of  an  Air  Command  and  Staff  College  report  cited 
previously,  provides  a  summary  of  the  benefits  and  problems  of 
the  CPOs  in  his  research  report.  According  to  Major  Luethemeier, 
the  CPOs  "demonstrate  the  command's  view  of  realism,"  thus  are  a 
beneficial  basis  for  all  training  and  evaluation.  In  addition, 
the  standardization  and  training  divisions  benefit  from  improved 
relations  because  of  the  common  "focus"  provided  by  CPO  standards 
(14:33).  On  the  other  hand,  Major  Luethemeier  cites  missile  unit 
dissatisfaction  with  specific  Level  4  time  standards  as  a  problem 
of  the  CPOs.  Because  Level  4  standards  are  the  results  of  a 
"best  estimate"  of  missile  experts  without  additional  testing, 
some  are  felt  to  be  "incorrect  or  inappropriate  time  periods." 
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There  is  also  concern  that  the  standards  "inhibit  trainer/ 
evaluator  judgement;  distract  crew  members;  and  convey  the  wrong 
signal,  emphasizing  speed  rather  than  correct  action"  (14:35). 

Though  other  specific  information  on  missile  units  is 
scarce,  a  brief  examination  of  the  benefits  and  problems 
surrounding  the  application  of  ISD  at  training  units  may  provide 
insight  into  similar  circumstances  at  the  missile  units. 

Benefits.  The  continuing  appeal  of  ISD  to  the  Air  Force 
stems  from  the  following  results  of  its  application: 

1.  Reduction  of  training  time.  Since  the  implementation  of 
ISD,  the  average  course  length  in  the  Air  Force  has  been  reduced 
from  16.8  weeks  in  1970  to  13  weeks  in  1978  (12:  3).  This  result 
stems  from  task  analysis  which  enables  instructors  to  identify 
and  train  only  necessary  tasks. 

2.  Reduction  of  costs.  Student  time  is  considered  to  be 
"the  largest  component  of  the  cost  of  technical  training" 

(12:20).  With  a  reduction  in  training  time,  student  time  has 
also  been  reduced.  Therefore,  training  costs  for  the  AF  peaked 
in  the  early  1970s  and  continued  to  drop  through  1979  (12:8). 

3.  Better  student  quality.  General  consensus  within  the 
training  profession  is  that  the  s tudent-cen teredness  of  ISD 
promotes  improvement  in  "student  performance,  motivation,  and 
learning  rates"  (12:63). 

4.  Flexible  guidelines  for  the  instructor.  ISD  models 
guide  instructors  through  a  systematic,  but  flexible  process. 
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They  enable  instructors  to  develop  training  programs  in  an 
orderly,  rather  than  haphazard  fashion. 

Problems.  Significant  problems,  however,  are  also 
encountered  by  training  units  in  the  application  of  ISD: 

1.  Time-consuming  training  development.  Though  course 
length  has  been  reduced,  the  time  spent  by  course  developers  has 
increased  since  the  implementation  of  ISD.  Task  analysis  and 
training  documentation  are  contributors  to  this  problem.  In  some 
cases,  it  has  been  necessary  to  spend  as  much  as  five  years 
developing  training  programs  (20:3).  One  resultant  tendency  has 
been  for  instructors  to  circumvent  the  process  by  developing 
shortcuts  (17:16).  There  is  also  some  tendency  to  apply  ISD  for 
only  a  portion  of  training  because  of  this  problem  (17:42). 

2.  Complex  nature  of  ISD.  ISD  is  systematic  and  flexible, 
but  application  of  any  ISD  model  requires  thorough  procedural 
knowledge.  Insufficient  knowledge  often  leads  to  overemphasis  or 
premature  selection  of  media  and  a  lack  of  proper  analysis, 
validation,  and  evaluation  (17:17).  An  obstacle  for  many 
instructors  is  the  development  of  concise,  understandable 
objectives  which  can  be  measured  (12:39).  Tests  are  also  an 
obstacle;  to  be  valid,  each  must  measure  the  specific  conditions, 
performances,  and  standards  specified  by  the  objectives  (12:91). 

3.  Management  support,  agency  cooperation.  Successful  ISD 
implementation  requires  close  management  attention.  Any  lack  of 
cooperation  or  support  between  agencies  hinders  the  process. 


Though  training  development  is  time  consuming  using  ISD, 
instructors  often  complain  about  "exhortations  from  higher 
headquar ters  to  get  on  with  it"  (12:84).  Constant  turnover  of 
military  personnel  results  in  instructors  who  lack  training  and 
experience  in  performing  ISD  tasks  (20:10).  ISD  teams  developing 
training  are  often  undermanned,  yet  team  members  are  required  to 
conduct  training  or  perform  additional  duties  at  the  same  time 
(17:  44-45).  There  are  also  "synchronization  problems";  various 
training  agencies  must  be  brought  into  the  process  at  different 
times  to  perform  different  tasks  (12:41). 

4.  Environmental  Impact.  Weapon  and  support  systems 
continuously  evolve  due  to  technological  advancement  and  problem 
resolution.  Procedural  changes  are  common  for  such  systems,  and 
technical  data  is  revised  often.  Training  programs  based  upon 
these  systems  therefore  encounter  a  "seemingly  endless  series  of 
changes  and  updates"  (18:330)  which  "send  reverberations 
throughout  ISD.  Revisions  to  audiovisual  material  can  consume  up 
to  60  percent  of  original  time  and  costs"  (17:30). 

5.  Student  problems.  Even  though  a  perception  of  enhanced 
student  quality  exists,  there  are  student-related  problems  with 
ISD.  "Many  students  find  it  difficult  to  adjust  immediately  to 
the  flexibility,  options,  and  increased  freedom"  of  ISD  training 
programs  (12:53).  More  significantly,  however,  has  been  the 
growing  concern  that  ISD-developed  training  programs  lack 
sufficient  theoretical  information  or  reinforcement  practice  to 


enable  long-term  retention  when  students  have  graduated.  The 
emphasis  on  student  achievement  of  specific  objectives  within  ISD 
enables  students  to  not  only  "absorb  the  material  very  quickly" 
and  "pass  a  test,"  but  also  "forget  it  very  quickly"  (20:8). 

Significance  to  Missile  Units .  It  is  not  specifically 
known  whether  missile  units  encounter  the  same  benefits  and 
problems  from  ISD  as  do  training  units.  The  previous  discussion, 
however,  provides  a  framework  for  later  measurement  of  the 
benefit  and  problems  of  ISD  at  the  missile  unit. 

Summary  and  Conclusion 

This  chapter  has  examined  numerous  sources  of  information  to 
provide  a  background  summary  on  the  Five-Step  and  CRI  ISD  models, 
as  well  as  the  potential  problems  and  benefits  of  such  models. 
Within  the  framework  of  this  background  summary  and  the  research 
questions  presented  in  the  previous  chapter,  a  data  collection 
survey  was  developed.  Development  of  the  survey  is  explained  in 
Chapter  III  of  this  research  project.  Other  background  material 
presented  in  Chapter  II  -  the  history  of  ISD  acceptance  within 
SAC,  and  the  identification  of  missile  unit  personnel,  programs, 
and  environments  -  formulated  a  basis  for  understanding  the  data 
colleted  by  the  survey.  Chapters  IV  and  V  analyze  and  draw 
conclusions  from  the  results. 
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Ill .  Methodolo 


Overview 

The  survey  method  was  used  in  this  thesis  to  collect 
specific  information  related  to  the  research  goal  and  questions. 
The  necessity  of  using  a  survey  for  this  purpose  stemmed  from  the 
lack  of  specific  literature  or  data  related  to  operational  units, 
especially  missile  units,  as  previously  described.  It  therefore 
is  the  purpose  of  this  chapter  to  detail  factors  relevant  to  the 
survey,  including:  the  population  sampled  by  it;  its 
construction,  refinement,  and  implementation;  the  method  of 
analyzing  survey  results;  and  limitations  of  the  survey. 

Survey  Sample 

The  implementers  of  ISD  at  the  missile  units,  those  persons 
who  develop  and  conduct  training  programs,  were  in  the  best 
position  to  provide  data  toward  the  goal  of  this  research.  For 
this  reason,  personnel  from  ISMD,  DOTI,  D022T,  and  D09  at  each  of 
the  eight  SAC  missile  units  were  selected  as  intended  recipients 
of  the  survey. 

The  original  intention  was  to  survey  the  entire  population 
from  these  divisions,  a  population  estimated  to  be  between  220 
and  240  persons.  As  the  methodology  for  this  research  was  being 
developed,  however,  Headquarters  (HQ)  SAC  began  its  own 
reassessment  of  the  CPOs.  It  feared  that  a  census  for  the 
purpose  of  this  research  report  could  lead  to  expectations  of 


change  on  the  part  of  those  surveyed,  thus  interfering  with  its 


reassessment.  This  concern  was  alleviated  by  reducing  the 
|  intended  number  of  survey  recipients  to  the  minimum  who  could 

provide  sufficient  data  to  enable  attainment  of  the  goal  of  this 
research . 

I 

TABLE  3.1 

Stratified  Sampling  Plan  for  This  Research 

Division  —  —  — —  — — 

Wing  ISMD  DOTI  D022T  D09 


44  SMW  1* 

(Ellsworth) 

9  0  SMW  1 

(F.E.  Warren) 

9 1  SMW  1 

( Minot ) 

308  SMW  1 

(Little  Rock) 

321  SMW  1 

(Grand  Forks) 

341  SMW  1 

( Malms trom) 

3  51  SMW  1 

(Whiteman ) 

38  1  SMW  1 

(McConnell ) 


111 
1  1  1  . 
Ill 

11** 

111 

111 

111 

11** 


survey  recipients  to  a  minimum.  Under  this  sampling  plan,  one 
key  person  within  each  training  division  at  each  unit  was 
surveyed.  A  key  participant  at  every  unit  was  the  Chief,  ISMD. 

As  the  coordinator  of  the  ISD  process  at  the  missile  unit,  the 
Chief,  ISMD,  provided  valuable  facts  and  opinions  regarding 
survey  topics.  Each  Chief,  ISMD,  was  also  requested  to 
recommend,  for  the  purpose  of  survey  participation,  the  names  of 
cadre  who  implemented  ISD  for  other  divisions.  With  this  method 
for  choosing  survey  participants,  there  was  relative  assurance 
that  facts  and  opinions  collected  would  provide  a  valid  picture 
of  the  ISD  process  at  the  missile  unit.  The  total  number  of 
participants  surveyed  under  this  plan  was  30. 

The  Survey  Instrument 

Communication  Modes.  Two  communication  modes,  telephone 
and  mail,  were  chosen  to  conduct  the  survey.  The  primary  mode  of 
communication  chosen  was  the  structured  interview,  conducted  by 
the  researcher  via  the  telephone.  This  choice  was  based  upon 
significant  advantages  it  offered  to  the  researcher,  including: 

A  great  deal  of  control.  In  fact,  telephone  interviews 
offered  the  greatest  degree  of  control  for  the 
researcher,  short  of  traveling  to  each  missile  unit  for 
face  to  face  interviews.  It  ensured  that  persons 
interviewed  would  not  misinterpret  questions  and  provide 
worthless  data. 

Development  of  a  dialogue  which  enabled  meaningful, 
two-way  communication. 
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Immediate  feedback  to  the  researcher,  rather  than  waiting 
for  results  which  could  sometimes  be  delayed  by  mail. 
Though  the  mail  survey  method  did  not  offer  any  of  the  above 
advantages,  it  was  chosen  as  a  backup.  It  was  recognized  that  in 
certain  instances,  persons  did  not  want  to  be  interviewed  by 
phone,  especially  when  they  were  constantly  busy  during  the  day 
or  when  working  in  a  division  where  classified  information  was 
typically  discussed  (i.e.  D022T  and  D09).  While  mail  surveys 
were  necessary,  they  were  limited  to  instances  such  as  those 
above  when  the  data  could  not  be  gathered  by  phone. 

Construction  of  the  Survey .  The  two  different 
communication  modes  necessitated  that  two  survey  instruments  be 
developed;  a  telephone  interview  schedule  (located  in  Appendix 
B)  and  a  mail  questionnaire  (located  in  Appendix  C).  Both 
instruments,  however,  were  similar  in  nature  -  their  survey 
questions  were  constructed  based  upon  five  considerations. 
Reliability,  the  "degree"  to  which  the  survey  questions  supply 
"consistent  results,"  was  one  consideration  (11:132). 

Reliability  was  promoted  by  asking  the  survey  questions  "the  same 
way,"  using  a  "standardized  format  and  sequence"  (11:215). 
Brevity,  a  second  consideration,  was  necessary  to  minimize  the 
additional  burdens  placed  upon  the  schedules  of  the  persons  being 
surveyed.  In  the  interest  of  brevity,  surveys  were  developed 
such  that  most  persons  would  complete  them  in  30-50  minutes, 
whether  by  phone  or  by  mail.  A  third  consideration  was  the 
introduction  to  the  survey.  It  was  carefully  designed  to 


identify  purpose  and  structure,  while  making  the  recipient  feel 
comfortable  in  participating.  Consideration  number  four  was  to 
gather  some  elements  of  background  information  from  each 
recipient.  Such  information  would  be  used  to  prevent  excessively 
broad  conclusions  about  data  gained  from  the  sample.  Background 
information  was  gathered  in  Part  I  of  both  the  interview  schedule 
and  the  questionnaire  (see  Appendices  B  &  C).  The  last 
consideration,  perhaps  the  most  important  one,  was  survey  content 
-  it  had  to  be  focused  toward  a  specific  purpose  in  order  to  be 
meaningful  for  this  research.  Survey  questions  were  therefore 
developed  using  information  from  Chapter  II  of  this  research 
project  as  the  basis  for  specific  measurement  of  the  research 
goal  and  questions  in  Chapter  I.  This  purposeful  development  is 
demonstrated  in  Table  3.2.  There  was  no  reason  to  believe  that 
any  particular  order  of  questions  was  necessary;  therefore, 
survey  questions  were  ordered  in  the  same  manner  as  the  research 
questions  presented  in  Chapter  I. 


TABLE  3.2 


How  Survey  Questions  Focus  Upon  The 
Original  Research  Goal  and  Questions 


Research  Goal 


Develop  a  detailed  description  of  ISD  as  it  relates  to 
developing  and  conducting  job  training  for  MCCMs  at  operational 
SAC  missile  units. 

Research  Question  #1 

Is  any  particular  ISD  model,  combination  of  models,  or 
component  parts  used  at  the  missile  unit? 

Measured  by  survey  questions  in  Parts  II  and  III  of  both 
the  telephone  interview  and  the  mail  questionnaire. 

(Refer  to  Appendices  B  and  C  for  verification.) 

The  "ISD-Related  Methodologies"  section  in  Chapter  II  of 
this  research  project  was  used  to  build  these  survey 
questions . 

Research  Question  #2 

What  are  the  perceived  problems/benefits  of  using  ISD  to 
develop  and  conduct  job  training? 

Measured  by  survey  questions  in  Parts  IV  and  V. 

The  "Usefulness  of  ISD"  section  in  Chapter  II  was  used  to 
build  these  survey  questions. 


Further  Exami nation  of  the  Survey  Questions .  Responses 
were  solicited  in  four  ways:  through  open-ended  questions, 
checklist  questions,  questions  with  four-point  scales,  and 
questions  with  five-point  scales.  Open-ended  questions  usually 
suggested  that  the  respondent  identify  examples  of  applications 
at  the  missile  unit  which  corresponded  to  specific  elements  of 
ISD  theory.  Respondents  were  permitted  to  provide  multiple 
examples  of  such  applications.  Telephone  interview  schedule 
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question  numbers  4,  6,  10,  13,  16,  17,  26,  and  29  were  open- 
ended,  as  demonstrated  in  Appendix  B  (numbers  6,  8,  12,  15,  18, 
19,  28,  and  31  in  the  mail  questionnaire  of  Appendix  C). 

A  checklist  format  was  used  as  a  second  method  of  response 
solicitation.  Telephone  interview  schedule  questions  1,  2,  18, 
27,  and  28  (and  3  ,  4,  20,  29  ,  and  30  of  the  mail  questionnaire) 
used  this  format.  Each  of  these  questions  suggested  response 
categories  from  which  the  respondent  could  choose.  Like 
open-ended  questions,  checklist  format  questions  usually  allowed 
multiple  responses,  as  well  as  the  opportunity  for  the  respondent 
to  provide  information  not  already  listed  on  the  survey. 

Four-point  scales  provided  yet  another  way  of  soliciting 
responses.  It  was  the  objective  of  these  scales  to  obtain  an 
estimate  of  how  often  particular  steps  or  aspects  of  ISD  theory 
were  used  at  the  unit.  In  all  but  one  question  involving  four- 
point  scales,  respondents  were  instructed  to  choose  from  the 
followi ng : 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


Questions 

3,  5,  7,  8,  9,  11, 

14, 

15, 

19, 

and 

20  of  the  telephone 

interview 

schedule  (5,  7,  9, 

10, 

11, 

13, 

16, 

17,  21,  and  22  of 

the  mail  questionnaire)  used  this  particular  four-point  scale.  A 
variation  of  this  scale  was  used  in  question  12  of  the  telephone 


interview  schedule  (number  14  in  the  mail  questionnaire). 
Potential  responses  listed  in  this  question  were: 

a.  Student  self -pacing  is  used  more  than  lecture 

b.  Student  self -pacing  and  lecture  are  used  equally 

c.  Lecture  is  used  more  than  student  self -pacing 

d.  Don 1 1  Know 

Both  of  these  four-point  scales  implied  a  rank  order  of  the 
response  alternatives;  alternative  "a"  in  each  of  these  scales 
ranked  closest  to  recommendations  of  ISD  theory,  while  the 
opposite  was  true  for  alternative  "c."  Each  of  these  scales  also 
implied  a  continuum  between  alternatives  "a"  and  "c,"  but  forced 
the  respondent  to  choose  the  alternative  which  most  described  his 
thoughts  on  how  often  a  particular  theoretical  aspect  of  ISD  was 
applied.  For  example,  there  were  instances  when  a  respondent 
replied  "b.  Sometimes"  to  a  question,  but  clarified  this  response 
with  an  additional  comments  such  as  "seldom."  Unlike  the  other 
alternatives,  there  were  nc  implications  of  rank  or  continuum 
with  alternative  "d.  Never."  Alternative  "d"  was  nevertheless 
included  to  accommodate  the  fact  that  some  persons  were  not 
totally  knowledgeable  about  ISD  applications  at  their  unit. 

The  final  manner  of  response  solicitation  was  through  five- 
point  scales.  Five-point  scales  were  used  to  measure  the 
strength  of  a  respondent's  opinion  regarding  benefits  and 
problems  of  the  CPOs.  The  opinions  of  the  respondents  were 
solicited  in  accordance  with  the  following  continuum: 

a.  Strongly  Agree 
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c.  Uncertain 


d.  Disagree 

e.  Strongly  Disagree 

This  continuum  was  used  in  the  construction  of  questions  21 
through  25  of  the  telephone  interview  schedule  (23  through  27  of 
the  mail  questionnaire). 

Refinement  and  Validation  of  the  Survey 

Upon  construction,  the  survey  underwent  a  refinement 
process.  This  refinement  process,  designed  to  ensure  its 
effectiveness  as  a  measurement  instrument,  began  with  a  survey 
tryout  on  three  former  developers  of  missile  training  at  the  Air 
Force  Institute  of  Technology.  Feedback  was  obtained  on 
particular  aspects  such  as  wording  and  order  of  questions  within 
the  interview  schedule. 

Validity  was  another  important  consideration;  the  interview 
schedule  had  to: 

"provide  adequate  coverage  of  the  topic  under  study" 

( 11:129) . 

ensure  that  its  measurements  "reflected  true  differences 
among  those  being  tested"  (11:128). 

For  instance,  did  measurement  of  problems /benefits  ensure 
adequate  coverage,  and  at  the  same  time,  reflect  true  differences 
in  ISD  usefulness  at  that  unit?  Validity  was  tested  in  two  ways: 
first,  the  interview  schedule  was  reviewed  by  three  ISD  experts 
at  HQ  SAC;  then,  a  trial  survey  was  conducted  at  the  381st  SMW, 


McConnell  AFB,  Kansas,  on  eight  persons  similar  to  those  actually 
surveyed  later.  While  the  survey  was  judged  valid  by  everyone 
who  participated  in  the  tryout,  the  process  resulted  in  a  few 
grammatical  changes  to  the  mail  questionnaire,  and  the  insertion 
of  statements  in  both  survey  communication  modes  to  allow  for 
comments  by  the  participants  after  any  questions. 

Survey  Implementation 

Survey  inplementation  followed  the  refinement  process. 
Interviews  were  conducted  in  June  and  July  of  19  85.  Responses 
recorded  from  each  participant  are  shown  on  a  question-by¬ 
question  basis  in  Appendix  D.  The  names  of  persons  have  not  been 
explicitly  identified  in  the  appendix  in  order  to  preserve 
anonymity. 

Data  Analysis 

Though  actual  analysis  of  the  responses  listed  in  Appendix  D 
is  deferred  until  Chapter  IV,  an  explanation  of  analysis 
methodology  is  in  order.  For  the  purpose  of  this  research, 
analysis  was  a  relatively  simple  exercise.  Similar  responses  for 
each  question  were  grouped  together,  counted,  and  recorded  in  a 
tabular  column  under  the  heading  "Frequency."  In  addition,  the 
percentage  of  persons  responding  with  the  same  answer  for  each 
question  was  calculated.  Each  result  was  recorded  in  a  column 
headed  "Relative  Frequency."  This  tabular  combination  of 


columns,  known  as  a  "Frequency  Table"  and  depicted  in  Table  3.3, 
became  the  basis  for  analysis  of  each  survey  question/response. 


TABLE  3.3 


An  Example  of  a  Frequency  Table 
For  Telephone  Interview  Schedule  Question  21 
and  Mail  Questionnaire  Question  23 


Response 

Frequency 

Relative 

Frequency 

Strongly  Agree 

9 

30 

Agree 

19 

63 

Uncertain 

1 

3 

Disagree 

1 

3 

Strongly  Disagree 

0 

0 

Comments  regarding  a  particular  survey  question  were 
sometimes  directed  toward  the  mode  of  the  frequency  table  (the 
response  most  frequently  given  by  those  surveyed).  The  mode  was 
a  good  indicator  of  the  respondents'  opinions  or  methods  when 
there  was  a  relatively  unequal  distribution  of  responses  within 
the  frequency  table.  When  two  or  more  responses  were  nearly 
equal  in  frequency  for  the  same  survey  question,  however,  the 
significance  of  the  mode  as  an  indicator  declined.  It  was 
therefore  necessary  to  direct  comments  toward  more  than  one 
response  category  for  many  frequency  tables.  In  addition, 
comment  was  generally  required  for  each  response  in  the  frequency 
tables  based  upon  replies  to  open-ended  questions.  The  author  of 
this  research  report  judged  all  the  responses  to  open-ended 
questions  as  valid,  based  upon  his  own  experience  at  the  missile 
unit . 
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While  analyzing  every  individual  frequency  table  in  this 
manner,  there  were  also  occasions  in  which  it  was  beneficial  to 
analyze  combined  groups  of  frequency  tables.  This  was 
particularly  true  of  those  frequency  tables  pertaining  to 
research  question  one,  originally  stated  in  Chapter  Is 

Is  any  particular  ISD  model,  combination  of  models,  or 
component  parts  used  at  the  missile  unit? 

Thus,  the  frequency  tables  analyzed  in  groups  were  drawn  from 
"Part  II:  Five-Step  Model"  and  "Part  III:  Criterion  Referenced 
Instruction  Model"  of  the  survey. 

The  frequency  tables  analyzed  in  groups  were  further  limited 
to  those  containing  the  four-point  scales  discussed  previously, 
because  analysis  involved  calculation  of  a  mean  (average)  score. 
While  this  did  not  include  every  frequency  table  pertaining  to 
the  Five-Step  or  CRI  Models,  the  information  provided  by  those 
questions  with  four-point  scales  was  sufficient  to  demonstrate 
how  often  missile  unit  instructors  apply  theoretical  aspects  of 
each  model.  Mean  scores  were  calculated  to  make  inferences 
about:  application  of  individual  steps  within  the  Five-Step 
Model,  overall  use  of  that  model,  overall  use  of  the  CRI  Model, 
and  combined  use  of  the  two  models. 

The  calculation  of  mean  scores  began  by  assigning  an 
arbitrary  value  to  three  of  the  response  categories  within  the 
four-point  scales.  Table  3.4  shows  the  value  assigned  to  each 
response  category.  The  "Don't  Know”  response  category  was  not 
assigned  a  value,  because,  as  discussed  previously,  it  was  not  a 


valid  member  of  the  continuum  formed  by  the  other  responses.  For 
this  reason,  any  person  responding  with  a  "Don't  Know"  answer  was 
not  included  for  the  purposes  of  calculating  a  mean  score.  The 
number  of  persons  responding  to  any  other  response  category, 
however,  was  multiplied  by  the  value  assigned  to  that  category, 
then  divided  by  the  total  number  of  responses  to  obtain  the  mean. 
This  resultant  score  provided  a  simple  basis  for  verbal  comment 
about  groups  of  frequency  tables,  because  it  would  correspond 
closely  to  one  of  the  original  response  categories. 

TABLE  3.4 

Arbitrary  Rank  Assignments  For  Each  Response  Category 
In  the  Four-Point  Scales 

Scale  1  Scale  2  Value 


Often  Student  self-pacing  is  used  more  2 

than  lecture 

Sometimes  Student  self-pacing  and  lecture  are  1 

used  equally 

Never  Lecture  is  used  more  than  student  0 

self -pacing 

Don' t  Know  Don' t  Know  None 


Limi tations  of  This  Methodology 

The  relative  simplicity  of  the  analysis  technique  was  a 
definite  advantage  of  the  methodology  described  in  this  chapter. 
This  advantage,  however,  was  offset  by  three  limitations  which 
detracted  somewhat  from  the  overall  ability  of  the  research 
report  to  make  inferences: 


1.  The  four-point  scale  used  to  calculate  mean  scores 
lacked  an  absolute  term  such  as  "Always"  at  the  top  of  the  scale 
to  oppose  "Never"  at  the  bottom.  Because  this  term  would  have 
been  assigned  the  highest  value  at  the  top  of  the  scale,  its 
inadvertent  deletion  has  most  likely  caused  a  slight  reduction  in 
mean  scores.  The  responses  and  scores  most  likely  distorted  from 
this  deletion  will  be  noted  in  Chapter  IV. 

2.  The  lack  of  any  four-point  scales  for  questions 
involving  the  last  step  of  the  Five-Step  Model  has  hindered 
accomplishment  of  a  complete  mean  score  analysis  for  this  model. 
Unfortunately,  mean  score  analysis  was  recognized  as  a  useful 
technique  after  construction  and  implementation  of  the  survey. 

3.  The  survey  sampling  plan  did  not  include  enough 
participants  to  make  inferences  about  the  total  population  with 
any  statistical  confidence.  Statistical  tests,  for  instance, 
could  not  be  used  to  ascertain  whether  differences  in  survey 
response  could  be  attributed  to  differences  in  a  person's 
experience,  division,  unit,  or  level  of  ISD  training. 

Despite  these  three  limitations,  the  author  of  this  research 
report  is  confident  that  valid,  valuable  information  has  been 
obtained  using  the  methodology  described  in  this  chapter.  The 
third  limitation  has  been  overcome  to  a  large  extent  by  the  fact 
that  survey  participants  were  among  the  cadre  who  implemented  ISD 
at  the  units.  While  the  second  limitation  has  been  impossible  to 
overcome,  there  still  were  enough  four-point  scales  to  analyze 
most  of  the  Five-Step  Model  using  mean  score  analysis.  Finally, 


though  mean  scores  have  been  slightly  distorted  by  the  first 
limitation/  their  validity  as  overall  measures  of  the  use  of  ISD 
models  can  be  confirmed  using  logic  when  actual  results  are 
analyzed  in  Chapter  IV. 

Summary 

This  chapter  has  provided  details  of  a  survey  constructed  to 
answer  research  questions  posed  in  Chapter  I.  In  addition,  it 
has  described  a  method  of  analysis  for  responses  to  the  survey, 
based  mostly  upon  frequency  tables,  but  also  upon  mean  scores  for 
limited  groups  of  questions.  Finally,  it  has  listed  three 
important  factors  which  limited  the  ability  of  this  research 
report  to  make  inferences  to  a  certain  extent,  but  nevertheless, 
were  factors  which  did  not  distract  from  overall  achievement  of 
the  research  goal.  Chapter  IV  analyzes  actual  responses  to  the 
survey  based  upon  the  methodology  described  in  this  chapter. 


Overview 


Responses  obtained  from  each  survey  question,  as  recorded  in 
Appendix  D,  are  analyzed  in  this  chapter.  For  the  purpose  of 
analysis,  survey  question/responses  are  categorized,  like  the 
survey  instruments,  into  five  headings: 

Background  Information 
Five-Step  ISD  Model 

Criterion  Referenced  Instruction  (CRI)  Model 
Problems  and  Benefits  of  the  CPOs 
Problems  and  Benefits  of  ISD  in  General 
Questions  categorized  under  each  particular  heading  are  analyzed 
on  an  individual  basis.  The  analysis  of  each  question  begins 
with  a  frequency  table  which  summarizes  the  responses  to  that 
particular  question.  Below  the  frequency  table  on  each  page  of 
analysis,  the  question  is  restated  from  the  survey  and  comments 
pertaining  to  the  results  are  provided.  In  an  effort  to  promote 
brevity  within  the  chapter,  only  the  question  numbers  for  the 
telephone  interview  schedule  are  mentioned.  If  the  reader 
desires  to  refer  to  the  mail  questionnaire  at  any  point  during 
this  analysis,  its  question  number  can  be  found  by  adding  two  to 
the  appropriate  telephone  interview  schedule  question  number. 

Mean  score  analysis  is  also  presented  as  necessary  during 
discussion  of  the  Five-Step  and  CRI  models. 
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Background  Information 


Telephone  Interview  Question  1:  How  much  experience 
have  you  had  developing  or  conducting  instruction  at  your  unit? 

TABLE  4.1 

Frequency  Table  for  Telephone  Interview  Question  1 


Response  , 

Frequency 

Relative 
Frequen cy 

Less  than  6  month -5 

1 

3% 

At  least  6  months  but 
less  than  1  year 

2 

7% 

At  least  1  year  but  less 
than  1  1/2  years 

4 

13% 

At  least  1  1/2  years  but 
less  than  2  years 

3 

10% 

2  years  or  more 

20 

67% 

Two  thirds  of  the  survey  participants  responded  that  they 
had  two  or  more  years  of  developing  and  conducting  instruction  at 
the  missile  unit  (see  Table  4.1).  This  result  was  not 
surprising,  since  the  participants  were  among  the  cadre  in 
implementing  ISD  there.  It  was  interesting  to  note,  however, 
that  only  one  of  the  six  participants  from  the  Codes  Division 
(D09)  had  this  much  experience. 


Relative 

k  * 

H 

Response 

Frequency  Frequency 

• 

Locally  developed  unit  training 

28 

9  3% 

SAC  Missile  Instructor  Course 

19 

63% 

* 

Criterion  Referenced  Instruction  Training 
(CRIT)  Course 

8 

27% 

Mini-CRIT  Course 

5 

17% 

Wing  Instructional  Systems  Manager 

Training  Course 

7 

23% 

Academic  Instructor  School 

6 

20% 

ATC  ISD  Courses 

6 

20% 

ECI  Course:  Principles  and  Techniques  of 
Instruction 

5 

17% 

College  Courses 

2 

7% 

This  question  presented  a  checklist  of  possible  responses 
for  survey  participants,  but  also  allowed  them  the  opportunity  to 
provide  additional  information.  Response  to  the  checklist  item, 
"Locally  developed  unit  training"  was  almost  unanimous,  and 


nearly  two  thirds  of  the  respondents  have  taken  the  SAC  Missile 
Instructor  Course  as  shown  in  Table  4.2.  The  large  number  of 
responses  in  each  of  these  two  categories  is  probably  due  to  the 


fact  that  both  types  of  training  can  be  accomplished  at  the 
missile  unit.  The  extent  to  which  courses  offered  outside  the 
missile  unit  have  been  attended  by  survey  participants  is  not 
nearly  as  great;  however,  the  combined  total  of  persons  who  have 
attended  either  the  Criterion  Referenced  Instruction  Training 
Course  or  the  Wing  Instructional  Systems  Manager  Training  Course 
constitutes  50  percent  of  the  respondents.  The  significance  of 
combining  totals  from  these  two  training  categories  stems  from 
the  fact  that  both  courses  conduct  exactly  the  same  training  for 
approximately  three  weeks.  The  Criterion  Referenced  Instruction 
Training  Course  then  concludes  after  three  weeks  while  the  Wing 
Instructional  Systems  Manager  Training  Course  presents  other 
training  specific  to  the  duties  of  ISMD  personnel  (see  Appendix  A 
for  a  detailed  description  of  each  course). 


Fi ve-Step  ISP  Model 

Telephone  Interview  Question  Does  your  unit 

accomplish  task  analyses  similar  to  Step  1  in  order  to  produce  a 
list  of  the  knowledge,  skills,  behaviors  duties,  and  tasks 
required  of  missile  combat  crew  members? 

TABLE  4.3 

Frequency  Table  For  Telephone  Interview  Question  3 


Response 

Frequency 

Relative 

Frequency 

Often 

7 

23% 

Sometimes 

16 

53% 

Never 

7 

23% 

Don' t  Know 

0 

0% 

This  question  and  the  responses  shown  in  Table  4.3  began  the 
discussion  of  the  Five-Step  model's  first  step,  ANALYZE  SYSTEM 
REQUIREMENTS.  A  slight  majority  of  the  survey  participants 
responded  that  task  analysis  was  accomplished  only  sometimes. 

The  minority  of  participants  who  responded  otherwise  were  equally 
distributed  in  their  answers:  half  reported  that  task  analysis 
was  performed  often  while  the  other  half  claimed  that  it  was 
never  done.  Analysis  of  the  next  survey  question  provided 
additional  insight  to  responses  for  this  question. 
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any  product  within  your  unit  or  SAC  which  can  be  regarded  as  the 
result  of  Step  1?  If  so,  please  specify. 


TABLE  4.4 

Frequency  Table  for  Telephone  Interview  Question  4 


Response 


Relative 

Frequency  Frequency 


CPOs  and/or  JPRs  25  83% 

New  systems  or  procedures  introduced 
to  unit  (i.e.,  61 6A  SLFCS;  crew 

maintenance  of  launch  control  center)  6  20% 

Task  analysis  performed  on  an  existing 
training  program  (i.e..  Transition 

Training)  3  10% 


This  was  an  open-ended  question  which  permitted  multiple 
responses  from  survey  participants.  An  overwhelming  majority  of 
participants  replied  that  the  CPOs  and/or  the  JPRs  were  the 
documented  products  of  a  task  analysis  within  SAC  (Table  4.4 
above).  Since  a  comprehensive  task  analysis  had  already  been 
performed  for  them  by  HQ  SAC,  most  participants  saw  little  need 
to  accomplish  task  analysis  on  their  own.  The  need  for  task 
analysis  generally  arose  when  new  systems  or  procedures  were 
introduced  to  the  misf  le  unit,  or  upon  revamping  of  an  existing 
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Telephone  Interview  Question  5:  Does  your  unit 
accomplish  the  tasks  listed  in  Step  2  in  order  to  produce  a  set 
of  instructional  standards? 


TABLE  4.5 

Frequency  Table  for  Telephone  Interview  Question  5 


Response 


Often 


Sometimes 


Never 


Don' t  Know 


Frequency 


Relat ive 
Frequen cy 


This  question  was  designed  to  solicit  responses  about  the 
frequency  of  use  of  Step  2,  Define  Education  &  Traini ..4 
Requi rements .  Responses  to  the  question  were  evenly 
distributed  between  "Often"  and  "Sometimes,"  with  about  half  as 
many  participants  responding  "Never"  as  compared  to  the  other  two 
categories  (see  Table  4.5).  While  conclusive  inferences  cannot 
be  drawn  based  upon  this  distribution  of  responses,  such  a 
distribution  nevertheless  indicates  less  than  frequent 
accomplishment  of  Step  2  by  missile  unit  instructors,  especially 
if  the  "Sometimes"  and  "Never"  categories  are  combined.  Further 
confirmation  of  this  inference  might  be  in  the  fact  that  survey 
participants  from  ISMD,  the  group  generally  considered  within  SAC 


to  be  most  knowledgeable  of  the  ISD  process,  unanimously  replied 


that  Step  2  was  merely  accomplished  "Sometimes." 

Telephone  Interview  Question  6:  Are  you  aware  of 
any  product  within  your  unit  or  SAC  which  can  be  regarded  the 
result  of  Step  2?  If  so,  please  specify. 


TABLE  4.6 

Frequency  Table  For  Telephone  Interview  Question  6 


Response 

Frequency 

Relative 

Frequency 

CPOs  and/or  JPRs 

21 

70% 

Step  2  tasks  performed  on  new  systems  or 
procedures  introduced  to  unit  (i.e., 

61 6A  SLFCS ,  crew  maintenance  on  launch 
control  center) 

6 

20% 

Step  2  tasks  performed  on  existing  training 
program  (i.e.  Transition  Training  or 
programmed  text) 

3 

10% 

Local  training  regulations.  Plans  of 
Instruction,  or  lesson  plans 

7 

23% 

Other  SAC  Regulations  (i.e.,  SACR  50-16, 

Vol  I  or  SACR  50-21) 

2 

7% 

An  indication  of  the  on-going  nature  of  first  two  steps  of 
the  Five-Step  model  was  demonstrated  by  responses  to  this 
question.  Three  of  the  five  response  categories  in  Table  4.6  are 
identical  to  those  for  the  question  on  the  products  of  Step  1. 

The  implication  of  one  of  the  remaining  response  categories  is 
that  local  training  regulations.  Plans  of  Instruction,  or  lesson 


57 


plans  are  used  to  document  the  results  of  Step  2  when  it  is 
accomplished  at  the  unit.  SAC  also  documents  other  training 
requirements  in  its  regulations,  as  indicated  by  the  last 
response  category. 

Telephone  Interview  Question  7:  Does  your  unit  use 
written  objectives  as  the  basis  for  training? 

TABLE  4.7 

Frequency  Table  for  Telephone  Interview  Question  7 


Response 

Frequency 

Relat ive 
Frequency 

Often 

30 

100% 

Sometimes 

0 

0% 

Never 

0 

0% 

Don' t  Know 

0 

0% 

This  question  began  the  discussion  of  Step  3  of  the 
Five-Step  Model,  DEVELOP  OBJECTIVES  AND  TESTS.  A  unanimous 
response  was  obtained:  training  has  often  been  based  upon 
written  objectives  (see  Table  4.7).  An  additional  comment, 
"Always,"  was  provided  by  a  few  participants.  This  additional 
comment,  combined  with  the  unanimous  responses  led  the  author  of 
this  research  report  to  suspect  that  a  response  category  "Always" 
would  have  been  chosen  on  a  wide  basis  had  it  been  offered  as  an 
alternative . 
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Telephone  Interview  Question  8:  Written  tests  at 


your  unit  are  based  upon  specific  objectives. 

TABLE  4.8 

Frequency  Table  for  Telephone  Interview  Question  8 


Response 

Frequency 

Relative 

Frequency 

Often 

25 

8  3% 

Sometimes 

5 

17% 

Never 

0 

0% 

Don" t  Know 

0 

0% 

While  not  unanimous  in  their  opinions,  survey  participants 
overwhelmingly  chose  the  "Often”  category  once  again,  as  shown  in 
Table  4.8.  Once  again,  additional  comments  offered  by  the 
participants  led  to  a  suspicion  that  an  "Always"  category  would 
have  been  selected  by  many. 
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TABLE  4.9 

Frequency  Table  for  Telephone  Interview  Question  9 


Response 

Frequency 

Relative 

Frequency 

Often 

30 

100% 

Sometimes 

0 

0% 

Never 

0 

0% 

Don' t  Know 

0 

0% 

For  the  third  simultaneous  question,  it  is  suspected  than  an 
"Always"  response  category  would  have  been  widely  selected,  since 
100  percent  of  the  participants  actually  chose  the  "Often" 
response  (Table  4.9).  By  this  point  in  the  analysis,  it  is  also 
obvious  that  objectives  play  an  extremely  important  part  in 
developing  and  conducting  training  at  the  missile  unit. 
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Telephone  Interview  Question  10:  Please  cite 
examples  of  criterion  tests  used  by  your  unit  which  require 
actual  demonstration  of  the  student's  performance. 

TABLE  4.10 

Frequency  Table  for  Telephone  Interview  Question  10 


Response 

Frequen  cy 

Relative 

Frequency 

MPT  scripts 

28 

9  3% 

Remedial  or  Individual  improvement  test 

4 

13% 

Quarterly  EWO  test 

3 

10% 

Learning  center  scenario  (accomplish 
procedures  on  mock-ups) 

7 

23% 

EWO  tape  testing 

3 

10% 

Performing  procedures  on  some  on-site 
equipment  (i.e.  biopack  donning, 
weather  corridor  plotting,  platform 
raising  and  lowering,  and  escape 
hatch  opening) 

5 

17% 

MPT  scripts  are  regarded  as  criterion  tests  by  93  percent  of 
survey  participants,  because  the  scripts  require  students  to 
perform  actual  procedures  in  a  Launch  Control  Center  (LCC) 
simulator.  None  of  the  other  responses  shown  in  Figure  4.10  were 
offered  by  a  majority  of  the  participants  for  this  open-ended 
question;  however,  it  is  likely  that  all  are  valid  examples  of 
criterion  tests.  Learning  centers  at  most  missile  units  are 
mock-ups  of  the  LCC,  and  though  less  sophisticated  simulators 


than  the  MPT,  are  places  where  the  student  can  demonstrate 
accomplishment  of  certain  crew  procedures.  All  MCCMs  are  also 
required  to  demonstrate  competence  using  equipment  located  at  the 
launch  facility,  either  during  training  alerts  or  Standardization 
Evaluations,  or  both.  MCCMs  who  fail  to  demonstrate  competence 
at  any  particular  task  are  often  provided  remedial  practice 
performing  the  tasks  until  it  can  be  accomplished  correctly. 
Finally,  EWO  tape  testing  and  quarterly  tests  can  be  considered 
as  criterion  tests  which  require  actual  demonstration  of  the 
student's  performance:  tape  testing  because  students  are 
required  to  copy  and  decode  Emergency  Action  Messages  as  they 
would  at  the  launch  facility;  and  quarterly  tests  because  they 
perform  calculations  similar  to  those  accomplished  under  war 
cond itions . 

Telephone  Interview  Question  11 :  Does  your  unit 
make  an  effort  to  sequence  objectives  and  instruction  in  a  manner 
which  is  felt  to  best  promote  student  learning? 


TABLE  4.11 

Frequency  Table  for  Telephone  Interview  Question  11 


Response 

Frequency 

Relat  ive 
Frequency 

Often 

19 

63% 

Sometime  s 

11 

37% 

Never 

0 

0% 

Don' t  Know 


0 


0% 
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A  discussion  of  Step  4,  PLAN,  DEVELOP,  AND  VALIDATE 
INSTRUCTION,  began  with  this  question.  According  to  the 
responses  listed  in  Table  4.11,  all  survey  participants  have  made 
an  effort  to  comply  with  this  aspect  of  ISD  theory.  Two-thirds 
of  the  participants  claimed  to  examine  the  sequencing  of  objec¬ 
tives  on  a  regular  basis  when  developing  instruction.  The 
remaining  one-third  who  responded  that  sequencing  was  performed 
on  a  less  frequent  basis,  also  indicated  that  greater  compliance 
with  this  aspect  of  ISD  theory  was  preferred;  however,  con¬ 
straints  such  as  busy  schedules,  student  failures  and  the  corre¬ 
sponding  remedial  training,  and  regulations  requiring  all  JPRs  to 
be  trained  annually  kept  them  from  having  the  time  to  do  so. 

Telephone  Interview  Question  12:  (Participants  were 
asked  to  choose  a  statement  which  best  described  the  use  of  self¬ 
pacing  as  opposed  to  lecture  at  the  missile  unit.) 


TABLE  4.12 

Frequency  Table  for  Telephone  Interview  Question  12 


Response 

Frequency 

Relative 

Frequency 

Student  self-pacing 
more  than  lecture 

methods  are  used 

0 

0% 

Student  self-pacing 
equally 

and  lecture  are  used 

15 

50% 

Lecture  is  used  more  than  student 
self -pacing 

15 

50% 

Don' t  Know 

0 

0% 
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As  demonstrated  in  Table  4.12,  none  of  the  thirty 
participants  claimed  that  student  self -pacing  was  used  more  than 
lecture.  A  plausible  explanation  for  this  unanimous  negative 
response  was  offered  through  the  additional  comments  of  two 
participants:  one  commented  that  SAC  does  not  permit  much  self- 
paced  training,  since  its  regulations  stipulate  minimum  time 
periods  for  monthly  classroom  sessions;  the  other  participant 
hypothesized  that  the  critical  nature  of  an  MCCM's  job  means  that 
the  unit  cannot  wait  for,  nor  rely  on,  students  who  train  at 
their  own  pace.  Despite  these  two  comments,  self-pacing  has  been 
used  at  least  to  some  extent  at  the  units.  Half  of  the 
participants  replied  that  it  was  used  on  an  equal  basis  with 
lecture.  Comments  from  this  group  of  participants  indicated  that 
the  overall  training  program  at  the  unit  included  self-paced 
complex  packages  and  remedial  training,  in  addition  to  monthly 
classroom  lectures.  The  remaining  participants  apparently 
dwelled  upon  the  monthly  classroom  sessions  when  they  replied 
that  lecture  was  more  common  than  student  self -pacing. 


Te le phone  Interview  Question  13 :  Please  cite 
examples  of  methods  in  which  student  participation  is  encouraged 
within  the  training  programs  of  your  unit. 


TABLE  4.13 

Frequency  Table  for  Telephone  Interview  Question  13 


Response 

Frequency 

Relative 

Frequency 

Guided  Question/Answer  Session 
during  monthly  classroom  training 

18 

60% 

Seminars  (i.e.  Transition  training 
cross  talks) 

2 

7% 

Student  critiques 

2 

7% 

MPT  Scripts 

3 

10% 

Learning  Center  mock-ups 

3 

10% 

EWO  Message  copying  exercises 

4 

13% 

Programmed  texts 

3 

10% 

Self-improvement  programs 

4 

13% 

Missile  complex  self-study  packages 

2 

7% 

Only  one  response  was  offered  by  a  majority  of  the 
participants  to  this  open-ended  question.  As  indicated  in  Table 
4.13,  many  of  the  missile  units  have  attempted  to  reduce  the  lack 
of  student  motivation  and  learning  generally  associated  with 
lectures  by  using  guided  question/answer  sessions  during  monthly 
classroom  training.  The  attempt  has  included  presentation  of 
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scenarios  typical  to  those  encountered  by  MCCMs  at  the  launch 
facility,  with  instructors  pausing  at  various  points  to  ask 
students  what  their  required  actions  would  be  in  the  situation. 

The  remaining  responses  listed  in  Table  4.13,  though  offered 
by  a  minority,  are  also  valid  examples  of  methods  used  to 
encourage  student  participation.  Seminars,  like  guided 
question/answer  sessions,  encourage  students  to  do  more  than  sit 
and  listen  during  classroom  training.  Programmed  texts, 
self-improvement  sessions,  missile  complex  self-study  packages, 
and  student  critiques  encourage  student  participation  through 
self-initiation  or  self-pacing.  Finally,  maximum  student 
involvement  occurs  when  MCCMs  demonstrate  competence  by  actually 
performing  tasks,  such  as  they  do  during  MPT  scripts,  learning 
center  mock-up  scenarios,  and  EWO  message  copying  exercises. 

Telephone  Interview  Question  14:  Does  your  unit 
evaluate  the  advantages  and  disadvantages  of  various  media  when 
developing  instruction? 

TABLE  4.14 

Frequency  Table  for  Telephone  Interview  Question  14 


Response 

Frequency 

Relat ive 
Frequency 

Often 

10 

33% 

Sometime  s 

20 

67% 

Never 

0 

0% 

Don' t  Know 


0 


0% 


The  Step  4  recommendation  regarding  media  evaluation  has  not 
been  followed  on  a  frequent  basis  by  missile  unit  instructors. 
Twenty  respondents  reported  that  media  evaluation  was 
accomplished  only  "Sometimes"  (Table  4.14).  Half  of  these  twenty 
respondents  provided  additional  clarifying  remarks.  Seven 
persons  remarked  that  limitations  such  as  consistently  short 
deadlines  for  getting  training  programs  developed  and  the  lack  of 
an  extensive  audio-visual  capability  at  the  unit  kept  media 
evaluations  to  a  minimum.  Three  other  respondents  candidly 
replied  that  the  medium  selected  for  inclusion  in  training  was 
that  which  was  convenient  or  had  been  used  before. 

Telephone  Interview  Question  15 :  Does  your  unit 
perform  a  tryout  on  a  small  group  of  students  in  order  to 
validate  instruction? 


TABLE  4.15 


Frequency  Table 

for  Telephone  Interview 

Question  15 

Response 

Frequency 

Relat ive 
Frequency 

Often 

0 

0% 

Sometimes 

15 

50% 

Never 

15 

50% 

Don ' t  Know 


0 


0% 


Instruction  has  been  validated  at  the  missile  unit  through 


tryout  on  a  small  group  of  students  only  to  a  limited  extent.  As 
demonstrated  in  Table  4.15,  there  was  an  even  distribution  of 
responses  in  the  "Sometimes"  and  "Never"  categories,  with 
comments  such  as  "Seldom,"  "Not  Practical,"  or  "Time  Doesn't 
Allow"  added.  Only  two  respondents  offered  examples  to  support 
their  belief  that  small  group  tryouts  occured  at  all:  one 
reported  that  MPT  scripts  at  his  unit  were  validated  through 
tryout  on  a  crew  of  non-instructors;  another  described  the  recent 
tryout  by  students  on  a  training  program  for  new  crew  maintenance 
procedures . 

Telephone  Interview  Que stion  16 :  Please  cite  any 
other  method  you  would  consider  a  means  of  validating 
instruction . 


TABLE  4.16 

Frequency  Table  for  Telephone  Interview  Question  16 


I 


Response 


Relative 

Frequency  Frequency 


Tryout  of  instruction  by  other 


instructors 

18 

60% 

Coordination  process 

13 

43% 

Analysis  of  test  or  evaluation  results 

6 

20% 

Student  critique  sheets  or  other 
student  feedback 


4 


13% 


The  survey  participants  have  indicated  a  variety  of 
plausible  validation  methods  as  shown  in  Table  4.16.  According 
to  almost  a  majority  of  survey  participants,  technical  accuracy 
has  been  validated  through  the  coordination  process  which 
requires  all  training  lesson  plans  to  be  reviewed  by  DOV  and 
staff  personnel.  A  greater  number  of  participants  indicated  that 
instructor  tryouts  of  training  programs  were  also  a  validation 
method,  because  they  ensured  training  programs  ran  as  planned. 
Analysis  of  test  or  evaluation  results  and  student  feedback  were 
offered  as  yet  other  methods  of  validation  by  a  minority  of 
participants,  because  these  methods  enabled  instructors  to  ensure 
students  could  achieve  training  objectives. 

Telephone  Interview  Question  17 ;  How  does  your  unit 
conduct  internal  evaluation  on  its  training  programs? 

TABLE  4.17 

Frequency  Table  for  Telephone  Interview  Question  17 


Relat ive 

Response 

Frequency 

Frequen  cy 

Analysis  of  Test  Results 

DOV  standardization  checks  on  students 

Periodic  formal  review  by  instructors 
or  staff 

Student  feedback 
Coordination  process 


This,  the  last  of  the  open-ended  questions  on  the  Five-Step 
model,  began  the  discussion  of  Step  5,  CONDUCT  AND  EVALUATE 
INSTRUCTION.  The  similarity  between  internal  evaluation  and 
validation  in  the  minds  of  missile  unit  instructors  has  been 
demonstrated  by  nearly  identical  responses  for  this  question  and 
the  last,  though  the  frequency  of  response  differed  between  each 
question.  Probably  all  responses  were  valid  examples  of  internal 
evaluation,  but  analysis  of  test  results  and  DOV  standardization 
checks  have  obviously  been  relied  upon  to  the  greatest  extent,  as 
demonstrated  in  Table  4.17. 

Telephone  Interview  Question  18 :  Which  would  you 
consider  external  evaluation? 

TABLE  4.18 

Frequency  Table  for  Telephone  Interview  Question  18 


Response 

Frequency 

Relative 

Frequency 

Standard izat ion/Training  Conferences 

16 

53% 

CPO  Conferences 

16 

53% 

EWO  Conferences 

12 

40% 

Higher  Headquarters  inspections 

27 

90% 

Staff  assistance  visits 

9 

30% 

Codes  conferences 

5 

17% 

Local  exercises 

7 

23% 

Comments  from  staff  officers 

3 

10% 

DOV  checks  on  crews 

3 

10% 

The  first  four  response  categories  shown  in  Table  4.18  were 
presented  as  a  check-list  of  items  requiring  a  simple  yes-no 
answer.  Three  out  of  the  four  categories  were  selected  by  a 
majority  of  respondents.  There  was  nearly  unanimous  approval 
that  Higher  Headquarters  Inspections  from  the  SAC  IG  and  3901st 
SMES  were  external  evaluation.  S  t  and  ar  d  i  zat  ion /Training 
Conferences,  designed  as  a  forum  for  discussion  of  problems  and 
potential  inprovements  for  missile  unit  training  programs,  and 
CPO  conferences,  convened  for  similar  purposes  regarding  the 
CPOs,  were  selected  by  16  out  of  30  participants  as  sources  of 
external  evaluation.  Though  a  majority  of  the  overall 
participants  did  not  select  EWO  conferences,  six  out  of  eight 
D022T  participants  did,  indicating  that  the  persons  for  whom  EWO 
conferences  cure  designed  consider  it  as  a  source  of  external 
ev  aluat  ion. 

Respondents  offered  five  other  plausible  examples  of 
external  evaluation  (listed  as  the  last  five  response  categories 
on  Table  4.18).  Though  none  of  the  esanples  were  offered  by  more 
than  30  percent  of  the  par  tic  ipan  ts ,  five  out  of  six  D09 
participants  included  Codes  Conferences  in  their  responses.  The 
inference  based  upon  these  responses  is  obviously  identical  to 
the  one  concerning  D022T  participants  and  EWO  conferences. 

Mean  Score  Analysis  for  the  Five-Step  Model .  As  discussed 
in  Chapter  III,  mean  score  analysis  was  used  to  summarize  how 
often  missile  unit  training  complied  with  the  recommendations 


of  ISD  theory.  This  summary  is  provided  in  Table  4.19.  Mean 
scores  for  Steps  1,  2,  &  4  indicated  less  than  frequent 
compliance  with  recommendations.  By  way  of  explanation,  survey 
participants  felt  little  need  for  accomplishment  of  Steps  1  &  2, 
because  it  was  their  common  belief  that  HQ  SAC  had  previously 
accomplished  the  steps  for  them.  The  CPOs  and  JPRs  were  the 
results  of  Steps  1  &  2,  according  to  the  participants. 
Constraints  such  as  SAC  regulations,  the  criticality  of  the  work 
of  MCCMs ,  deadlines,  and  the  lack  of  audio-visual  resources  were 
felt  to  interfere  with  the  procedures  described  in  Step  4. 


TABLE  4.19 

Mean  Score  Analysis  of  the  Five-Step  Model 


Step(  s ) 

Mean  Score 

Corresponding  To 

1 

1.00 

Sometimes 

2 

1.17 

Sometimes 

3 

1.94 

Often 

4 

0.99 

Sometimes 

5 

None  Calculated 

- 

1-4 

1.33 

Sometimes 

Unlike  mean  scores  calculated  for  the  other  steps,  those 
calculated  for  Step  3  indicated  frequent  compliance  with  its 
recommendations.  The  nearly  unanimous  response  from  the 
participants  indicated  that  lesson  plans,  tests,  and  MPT  scripts 
were  based  upon  objectives  "Often."  As  discussed  previously, 


there  was  even  possibility  that  an  "Always"  category  would  have 
been  chosen  to  a  great  extent. 

Implications  of  this  possibility  are  that  both  the  mean 
score  for  Step  3  and  the  overall  mean  score  are  lower  than  if  an 
"Always"  category  had  been  included.  Thus,  the  overall  mean 
score  of  1.33  is  perhaps  slightly  low.  Though  this  score  may  not 
be  totally  valid,  there  is  little  possibility  that  a  score  based 
upon  better  response  categories  would  increase  beyond  1.5, 
because  270  responses  were  used  in  the  overall  mean  calculation. 

A  less  than  frequent  compliance  with  the  recommendations  of  the 
Five-Step  Model  is  thus  the  overall  inference  based  upon  mean 
score  analysis. 

Cri  terion  Referenced  Instruction  (CRI )  Model 

Telephone  Interview  Question  19 ;  Does  your  unit 
perform  goal  analysis  when  developing  training  programs? 

TABLE  4.20 

Frequency  Table  for  Telephone  Interview  Question  19 

Relat ive 

Response  Frequency  Frequency 

Often  2  7% 

Sometimes  22  7  3% 

Never  3  10% 


Don' t  Know 


3 


10% 


Seventy-three  percent  of  survey  participants  responded  that 


goal  analysis,  as  prescribed  by  Dr.  Robert  F.  Mager  in  the  CRI 
model,  was  performed  "Sometimes"  (see  Table  4.20).  Of  this  73 
percent,  five  participants  clarified  their  response  with  a 
"Seldom"  comment.  Another  six  persons  in  that  group  said  that 
though  goal  analysis  was  performed,  it  was  accomplished  purely  a 
common  sense,  not  as  the  explicit  analysis  recommended  by  the 
model . 

Telephone  Interview  Question  20 ;  Does  your  unit 
present  instruction  in  the  manner  recommended  by  Mager? 

TABLE  4.21 

Frequency  Table  for  Telephone  Interview  Question  20 


Response 

Frequency 

Relat ive 
Frequency 

Often 

8 

27% 

Sometimes 

17 

57% 

Never 

5 

16% 

Don' t  Know 

0 

0% 

This  question  was  preceded  by  a  brief  explanation  of  Dr. 
Mager' s  recommendation  for  conducting  instruction.  His 
recommendation  involved:  the  presentation  of  objectives  for  the 
student  to  achieve;  the  identification  of  sources  of  information 
about  the  objectives;  provisions  for  student  practice  in 
achieving  the  objectives;  and  criterion  tests  as  a  measure  of 


their  achievement.  Fifty-seven  percent  of  the  participants 
responded  that  instruction  at  their  unit  was  presented  in  this 
manner  at  least  "Sometimes,"  with  another  27  percent  saying  it 
was  performed  "Often"  (Table  4.21).  Examples  of  unit  training 
conducted  in  this  manner  were  enrichment  or  remedial  programs  at 
two  of  the  units,  and  missile  complex  study  packages  at  another 
four. 

Mean  Score  Analysis  for  the  CRI  Model .  Application  of 
the  CRI  Model,  like  the  Five-Step  Model,  was  limited.  Mean 
scores  for  the  unique  aspects  of  the  CRI  Model  hovered  in  the 
"Sometimes"  range,  as  demonstrated  in  Table  4.22. 


TABLE  4.22 

Mean  Score  Analysis  of  the  CRI  Model 


Category 

Mean  Score 

Corresponding  To 

Goal  Analysis 

0.96 

Sometimes 

Mager  Method 

1.10 

Sometimes 

Overall  Model 

1.04 

Sometimes 

Mean  Score 

Analysis  for  Both  Models. 

The  combined  mean 

score  for  the  two  models  was  1.28,  no  surprise  based  upon 
previous  discussions. 


beneficial  basis  for  training  and  evaluation 


TABLE  4.23 

Frequency  Table  for  Telephone  Interview  Question  21 


Response 

Frequency 

Relative 

Frequency 

Strongly  Agree 

9 

30% 

Agree 

19 

63% 

Uncertain 

1 

3% 

Disagree 

1 

3% 

Strongly  Disagree 

0 

0% 

There  was  most  definite  agreement  among  survey  participants 
that  the  CPOs  were  a  useful  instrument  at  the  missile  unit  (refer 
to  Table  4.23).  Ninety-three  percent  of  the  survey  participants 
showed  their  support  for  the  above  statement,  although  most  were 
not  willing  to  give  it  their  strongest  support. 


Telephone  Interview  Question  22:  Improved  communication 
between  DOV  and  the  training  divisions  is  one  benefit  of  the  CPOs 


TABLE  4.24 

Frequency  Table  for  Telephone  Interview  Questtion  22 


Response 


Frequency 


Relative 

Frequency 


Strongly  Agree 
Agree 
Uncertain 
Disagree 

Strongly  Disagree 


and  appropriate  time  standards. 


Response 

Frequency 

Relat ive 

Frequen cy 

- 

I 

Strongly  Agree 

1 

3% 

Agree 

15 

50% 

• 

Uncertain 

8 

27% 

4 

Disagree 

4 

13% 

Strongly  Disagree 

2 

7% 

evaluator  judgement  on  the  MPT. 


TABLE  4.26 

Frequency  Table  for  Telephone  Interview  Question  24 


Response 

Frequen  cy 

Relative 

Frequency 

Strongly  Agree 

8 

27% 

Agree 

11 

37% 

Uncertain 

1 

3% 

Disagree 

7 

23% 

Strongly  Disagree 

3 

10% 

Support  for  this  negative  statement  about  the  CPOs  has  been 
indicated  by  survey  participants.  Individually,  each  of  the 
"Strongly  Agree"  and  "Agree"  response  categories  shown  in  Table 
4.26  were  chosen  by  more  participants  than  any  other  category. 
Combined  totals  for  these  two  categories  also  outnumbered  the 
remaining  responses  by  a  two-to-one  margin.  Participants  from 
DOT I  gave  the  greatest  support  to  this  statement;  out  of  eight 
persons  interviewed,  four  responded  "Strongly  Agree”  and  two 
"Agree."  D022  personnel  gave  slightly  more  reserved  support  for 
the  validity  of  the  statement;  three  out  of  the  eight 
participants  agreed,  while  another  two  strongly  agreed.  An 
additional  point  of  interest  about  responses  to  this  statement: 
while  the  statement  was  designed  to  examine  a  possible  problem  of 


the  CPOS/  two  participants  commented  that  inhibition  of  trainer/ 
evaluator  judgement  on  the  MPT  was  a  benefit,  because  much 
subjectivity  was  taken  out  of  their  hands  by  the  CPOs. 

Telephone  Interview  Question  25 ;  CPOs  convey  the 
wrong  signal  by  emphasizing  speed  rather  than  correct  action. 

TABLE  4.27 

Frequency  Table  for  Telephone  Interview  Question  25 


Response 

Frequen  cy 

Relat ive 
Frequency 

Strongly  Agree 

4 

13% 

Agree 

6 

20% 

Uncertain 

1 

3% 

Disagree 

17 

57% 

Strongly  Disagree 

2 

7% 

A  definite  majority  disagreed  with  this  statement  about  a 
potential  problem  with  the  CPOs.  The  combined  responses  within 
the  last  two  categories  in  Table  4.27  outnumbered  the  other 
categories  19  to  11,  a  two-to-one  margin  toward  disagreement. 

Telephone  Interview  Question  26 :  Are  there  any 
other  benefits  or  problems  with  the  CPOs  that  you  would  care  to 
d iscuss? 
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TABLE  4.28 


Frequency  Table  for  Telephone  Interview  Question  26 


Response 


Relat ive 

Frequency  Frequency 


Benefits 

Provides  basic  expectations  for  MCCMs  5 

Standardizes  training  &  evaluation  4 

Problems 

Students  learn  only  what  they  have  to  5 

CPOs  not  validated  prior  to  publishing  or 
updated  in  a  timely  fashion  4 

CPOs  inhibit  training  3 

CPOs  cause  problems  of  interpretation  3 

Not  enough  flexibility  and  realism  2 

Don' t  allow  all  required  actions  to  be 
evaluated  2 


17% 

13% 

17% 

13% 

10% 

10% 

7% 

7% 


There  was  no  single  response  offered  by  a  majority  of  the 
survey  participants,  indicating  that  the  most  common  benefits/ 
problems  of  the  CPOs  have  been  discussed  prior  to  this  open- 
ended  question  (refer  to  Table  4.28). 

Problems  and  Be ne f i t s  of  ISP  i n  General 

Telephone  Interview  Question  27 :  Which  statements 
reflect  your  opinion  of  the  benefits  of  using  ISD  at  the 
operational  missile  unit? 
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TABLE  4.29 


Frequency  Table  for  Telephone  Interview  Question  27 


Response 

Frequency 

Relat ive 
Frequency 

Student  performance  has  increased 

11 

37% 

Student  learning  rates  have  improved 

14 

4  7% 

Student  motivation  has  improved 

7 

23% 

Total  training  time  is  reduced 

12 

40% 

Costs  are  reduced 

8 

26% 

ISD  provides  definite,  yet  flexible 
guidelines  for  developing  and 
conducting  instruction 

26 

8  7% 

I  don*  t  see  any  benefits  in  using 

ISD  at  my  unit 

3 

10% 

Provides  a  command  focus  for  training 
and  evaluation 

2 

6% 

Makes  us  look  at  entire  training  program 
(  in  other  words  there  is  a  complete 
training  loop) 

2 

6% 

Only  one  item  in  Table  4.29  above  was  regarded  by  a  majority 
of  survey  participants  as  a  benefit  of  ISD  at  the  missile  unit. 

In  fact,  nearly  all  participants  regarded  ISD  theory  to  be  a 
useful  outline  for  developing  and  conducting  training.  Many  of 
the  participants  were  reluctant  to  identify  any  other  benefit 
because  they  had  not  been  at  the  missile  unit  prior  to  the 
introduction  of  ISD,  and  thus  had  no  basis  for  comparison  with 
other  training  methods.  Those  participants  who  did  identify 
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other  benefits,  however,  often  chose  each  of  the  first  five 
response  categories  in  Table  4.29.  Thus,  it  is  quite  possible 
that  all  five  of  these  categories  are  benefits,  despite  the  lack 
of  overwhelmi ng  support  for  them. 

Telephone  Interview  Question  28 ;  Which  statements 
reflect  your  opinions  of  the  problems  caused  by  using  ISD  at  the 
operational  missile  unit? 


L25  Vlll.4 


TABLE  4.30 


Frequency  Table  for  Telephone  Interview  Question  28 


Response 


Relative 

Frequency  Frequency 


It  is  extremely  time  consuming  to 
develop  training  using  ISD  14 

Developing  concise,  understandable 
objectives  is  a  difficult  task  for 
many  persons  IS 

Lack  of  cooperation  or  support  between 
agencies  hinders  the  process  21 

Constant  turnover  of  military  personnel 
results  in  instructors  who  lack 
training  and  experience  in  performing 
ISD  tasks  28 

Higher  Headquarters  regulations  and 
directives  often  interfere  with  the 
process  15 


The  seemingly  endless  series  of  changes 
to  the  weapon  system,  regulations, 
and  technical  data  hinders  the  ISD 
process.  Much  time  and  money  is 
consumed  re-doing  tasks  16 

ISD-developed  training  programs  lack 
sufficient  theoretical  information 
to  enable  long-term  retention.  (In 
other  words,  students  can  absorb 
material  quickly,  pass  a  test, 

then  forget  it  quickly.)  19 

I  don't  see  any  problem  using  ISD  at  my 
unit  0 


47% 

60% 

70% 

9  3% 

50% 

53% 

63% 

0% 


Students  don't  care  unless  task  is 
evaluatable  and  within  CPOs 


3 


10% 


Respondents  identified  more  problems  than  benefits  of  using 
ISD  at  the  missile  unit.  The  first  seven  response  categories  in 
Table  4.30  were  widely  recognized  as  problems.  While  little 
additional  information  was  provided  by  respondents  about  each  of 
these  problems,  each  has  received  a  detailed  explanation  in 
Chapter  II,  during  the  discussion  of  problems  at  training  (i.e. 
ATC)  units. 

Chapter  Summary/Conclusions 

Telephone  Interview  Question  29:  I  would  appreciate 
any  other  comments  regarding  this  survey,  ISD  in  general,  or  any 
specific  aspects  of  ISD. 


TABLE  4.31 

Frequency  Table  for  Telephone  Interview  Question  29 


Response 


Relative 

Frequency  Frequency 


ISD  good  in  theory,  not  totally 
applicable  at  missile  unit  8  27% 


Thirteen  participants  summarized  their  thoughts  on  the 
application  of  ISD  at  the  missile  unit  by  responding  to  this 
survey  item.  A  common  theme  developed  by  eight  of  them  was  that 
ISD  provided  interesting  theory,  but  was  not  totally  useful  at 
the  missile  unit.  According  to  these  participants,  missile  units 
have  not  been  permitted  to  follow  all  aspects  of  theory;  factors 
such  as  regulations,  HQ  inspections,  deadlines,  and  the  critical 
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nature  of  the  job  of  the  MCCM  limited  extensive  application. 
Therefore,  ISD  has  been  considered  merely  a  tool  by  these  eight 
participants  -  a  tool  modified  as  necessary  to  meet  unit  needs. 

This  chapter  has  analyzed  the  results  of  a  survey  on  this 
ISD  tool.  Responses  to  twenty-nine  questions  or  statements 
regarding : 

background  characteristics  of  the  participants, 

-  applications  of  the  Five-Step  and  CRI  Models,  and 
problems  and  benefits  of  the  CPOs  and  ISD  in  general 
have  been  examined.  Frequency  tables  were  used  to  a  great  extent 
in  this  chapter,  and  mean  scores  to  a  lesser  extent  as  a  basis 
for  analysis.  This  analysis  will  be  used  in  the  next  chapter  to 
develop  a  description  of  the  ISD  process  as  it  relates  to  the 
operational  missile  unit. 


V.  CONCLUSIONS 


Overview 

This  final  chapter  states  the  conclusions  reached  through 
this  research  effort.  Three  categories  of  conclusions  are 
reported:  a  description  of  ISD  as  it  relates  to  developing  and 
conducting  MCCM  job  trainiing;  an  identification  of  the 
implications  surrounding  the  specific  application  of  ISD  at  the 
missile  unit;  and  a  listing  of  recommendations  for  enhancement  of 
the  ISD  process.  Most  conclusions,  however,  are  reported  within 
the  first  category,  since  the  overall  goal  of  this  thesis  is  to 
provide  such  a  description. 

A  Description  of  ISD  at  the  Missile  Unit 

Two  research  questions,  posed  at  the  beginning  of  this 
thesis  have  served  as  the  basis  for  describing  ISD  as  it  is 
related  to  developing  and  conducting  MCCM  job  training: 

1.  Is  any  particular  ISD  model,  combination  of  models, 
or  component  parts  used  at  the  missile  unit? 

2.  What  are  the  perceived  problems/benefits  of  using 
ISD  to  develop  and  conduct  job  training? 

Plausible  answers  to  the  research  questions  have  been  found  by 
analyzing  survey  responses  from  a  stratified  sampling  of  30  SAC 
missile  unit  instructor  cadre.  Answers  to  the  research  questions 
are  provided  below. 


1.  Is  any  particular  ISP  model ,  combination  of 
models,  or  component  parts  used  at  the  missile  unit? 

Analysis  of  survey  responses  demonstrates  that  two  ISD  models, 
the  Five-Step  Model  and  the  CRI  Model,  are  applied  to  a  limited 
extent  at  the  SAC  missile  unit.  Though  all  specific  procedures 
recommended  within  each  model  are  applied,  rarely  are  they 
applied  more  than  sometimes.  Instead,  recommended  procedures  are 
often  omitted  or  modified. 

The  first  two  steps  of  the  Five-Step  Model, 

-  ANALYZE  SYSTEM  REQUIREMENTS 

-  DEFINE  EDUCATION  AND  TRAINING  REQUIREMENTS 

are  most  commonly  omitted  when  developing  MCCM  job  training. 

Most  instructor  cadre  consider  the  JPRs  and  CPOs  as  products  of 
Steps  1  and  2.  As  comprehensive  listings  of  MCCM  tasks  and 
standards,  the  JPRs  and  CPOs  usually  preclude  the  need  for 
instructors  at  the  missile  unit  to  accomplish  task  analysis  or 
develop  instructional  standards  prescribed  by  the  model  at  these 
steps.  Specific  occasions  for  performance  of  these  steps  at  the 
missile  unit  are  relatively  few.  Generally,  Step  1  task  analysis 
is  performed  upon  introduction  of  new  systems  or  procedures 
affecting  MCCMs,  but  not  yet  covered  by  JPRs  or  CPOs.  The 
analysis  performed  by  Minuteman  units  on  new  MCCM  maintenance 
taskings  is  an  example  of  Step  1  accomplishment.  Very  rare  are 
occasions  in  which  task  analysis  is  performed  on  existing 


training  programs;  however,  Titan  units  have  performed  such 


analysis  on  the  job  o£  a  Missile  Combat  Crew  Commander,  in  an 
effort  to  improve  transition  training.  After  these  rare 
occasions  of  Step  1  accomplishment,  tasks  selected  for  training 
are  documented  in  local  training  regulations,  Plans  of 
Instruction,  or  supplemental  lesson  plans,  in  accordance  with 
Step  2. 

Recommendations  of  the  Five-Step  Model  are  more  frequently 
and  carefully  applied  at  Step  3,  DEVELOP  OBJECTIVES  AND  TESTS, 
than  any  other  step.  Unlike  the  other  steps.  Step  3  is  never 
omitted  or  modified.  It  is  the  beginning  of  the  ISb  process  at 
the  missile  unit  upon  most  occasions.  Written  objectives 
developed  at  this  step  specify: 

each  behavior  required  of  an  MCCM, 

-  the  conditions  under  which  each  behavior  must  be 
accomplished,  and 
the  standards  for  accomplishment. 

Objectives  are  the  basis  for  all  lesson  plans,  tests,  and  MPT 
scripts.  Criterion  tests,  also  developed  during  this  step, 
emphasize  actual  demonstration  of  an  MCCM's  achievement  of 
objectives.  Many  examples  of  such  criterion  tests  exist  at  the 
units,  according  to  instructor  cadre:  MPT  scripts,  remedial  or 
individual  improvement  tests,  learning  center  scenarios,  EWO  tape 
testing,  quarterly  EWO  tests,  and  demonstrations  of  competence  on 
on-site  equipment.  By  requiring  MCCMs  to  physically  demonstrate 


their  competence,  all  of  these  criterion  tests  correspond  closely 
to  the  recommendations  provided  within  Step  3  of  the  model. 

The  specific  recommendations  provided  within  Step  4,  PLAN, 
DEVELOP,  AND  VALIDATE  INSTRUCTION,  lack  widespread  application  at 
the  missile  unit,  with  one  exception.  Instructors  often  attempt 
to  sequence  objectives  and  instruction  in  a  manner  which  they 
feel  best  promotes  student  learning,  as  recommended  by  the  model. 
Unlike  model  recommendations,  however,  there  is  less  than 
frequent  evaluation  of  the  potential  media  to  be  used  for 
training.  Constraints,  such  as  short  deadlines  or  the  lack  of 
audio-visual  capability,  often  hinder  media  evaluations,  leaving 
instructors  to  choose  the  most  convenient  medium  or  the  one  which 
had  been  used  before. 

Self-paced  methods  of  instruction  are  not  used  as  much  as 
recommended  by  the  model  at  Step  4  either.  At  the  missile  unit, 
self-pacing  is  no  more  common  than  lecture,  possibly  even  less 
common.  There  are  relatively  few  programmed  texts  at  the  missile 
units,  and  the  only  self-paced  program  common  to  all  missile 
units  is  the  monthly  study  package  available  at  each  of  the 
missile  complexes.  The  critical  nature  of  the  work  of  MCCMs 
hinders  great  reliance  upon  self-pacing,  as  well  as  SAC  training 
regulations  which  specify  minimum  training  periods  for  all  MCCMs. 
Though  their  reliance  upon  self-pacing  is  less  than  recommended 
at  Step  4,  missile  unit  instructors  recognize  the  importance  of 
student  involvement  promoted  by  such  training.  Instructors 


therefore  substitute  a  variety  of  other  training  techniques  and 
methods  in  an  effort  to  promote  student  involvement.  Examples  of 
such  techniques  and  methods  include:  seminars,  student 
critiques,  MPT  scripts,  learning  center  scenarios,  EWO  message 
copying  exercises,  self-improvement  programs,  and  guided 
question/answer  sessions  during  lectures. 

The  recommendations  of  Step  4  regarding  validation  of 
training  programs  are  also  modified.  Seldom  is  instruction 
validated  through  tryouts  on  small  groups  of  students  prior  to 
the  starting  date  of  the  program.  Instead,  instructors  rely  upon 
the  formal  coordination  process  within  the  missile  unit  and  upon 
tryouts  by  other  instructors  to  ensure  training  material  is 
accurate.  Students  are  generally  brought  into  the  validation 
process  after  formal  training  has  started.  Evidence  of  their 
ability  to  meet  the  objectives  of  the  training  program  is  found 
in  their  test  and  evaluation  results.  Their  critiques  also 
provide  feedback  about  the  worthiness  of  training  developed  at 
this  step. 

By  the  time  student  validation  is  included,  missile  unit 
programs  have  moved  into  Step  5  of  the  model,  CONDUCT  AND 
EVALUATE  TRAINING.  Missile  unit  instructors  consider  student 
validation  to  be  identical  to  the  internal  evaluation 
recommended  by  this  last  step.  Thus,  a  large  portion  of 
validation  and  internal  evaluation  are  performed  concurrently. 
Missile  unit  instructors  also  rely  upon  periodic  formal  review  by 
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staff  members  within  the  Operations  Deputate  for  further  internal 
evaluation . 

External  evaluation  of  training  programs  at  Step  5  is 
generally  considered  to  occur  through  interaction  with  agencies 
outside  the  missile  unit.  Inspections  from  agencies  such  as  the 
SAC  IG  and  the  3901st  SMES  are  important  determinants  of  the 
ability  of  a  particular  missile  unit  to  train  MCCMs  to  accomplish 
tasks  under  conditions  and  standards  listed  in  the  CPOs.  These 
inspections  also  examine  the  appropriateness  of  particular  CPOs, 
thus  providing  important  feedback  to  HQ  SAC  regarding  possible 
revisions.  Similar  feedback  is  provided  through  periodic 
conferences  involving  all  missile  units.  Examples  of  such 
conferences  convened  under  the  sponsorship  of  HQ  SAC  include: 
Standardization/Training  conferences,  CPO  conferences,  EWO 
conferences,  and  Codes  conferences.  By  providing  feedback  for 
possible  revisions,  these  conferences  and  inspections  link  the 
final  step  of  the  Five-Step  Model  to  its  first  step. 

The  Five-Step  Model,  as  applied  at  the  operational  missile 
unit,  is  characterized  by  heavy  SAC  involvement  at  the  beginning 
and  the  end.  This  involvement  drastically  reduces  the  need  for 
application  of  Steps  1  and  2  by  missile  unit  instructors,  but 
keeps  them  participating  at  Step  5.  In  between  these  steps, 
there  is  widespread  application  of  the  recommendations  by  the 
model  at  Step  3,  and  modified  application  of  the  recommendations 
at  Step  4. 


Application  of  the  CRI  Model  is  no  more  extensive  than  that 
of  the  Five-Step  Model,  since  both  offer  similar  recommendations 


about  developing  and  conducting  training  programs.  Even  the 
unique  contributions  of  CRI  to  ISD  theory  find  less  than  frequent 
use  at  the  missile  unit.  There  is  lack  of  explicit  goal  analysis 
performed  at  the  unit,  for  instance.  On  the  rare  occasions  goal 
analysis  is  performed  for  training  purposes,  it  is  generally  an 
undocumented,  common  sense  process.  Instructors  find  somewhat 
greater  use  for  the  model's  recommendation  of  a  specific  method 
for  conducting  instruction  (involving  the  presentation  of 
objectives  for  the  student  to  achieve,  the  identification  of 
sources  of  information  about  the  objectives,  provisions  for 
student  practice  in  achieving  the  objectives,  and  criterion  tests 
as  measure  of  their  achievement).  Use  of  this  method  is  no 
greater  than  sometimes,  however. 

2.  What  are  the  percei ved  problems /be nef it s  of  usi nq 
ISD  to  develop  and  co nd uct  job  training?  Instructor  cadre 
identify  a  number  of  benefits  and  problems  regarding  the  general 
application  of  ISD  at  the  missile  unit.  There  is  overwhelming 
recognition  by  instructor  cadre  that  one  benefit  of  ISD  is  its 
definite,  yet  flexible  guidelines  for  developing  and  conducting 
instruction.  Comments  offered  by  missile  unit  cadre  indicate 
that  ISD  models  are  regarded  as  theoretical  tools  which  can  be 
modified  to  meet  the  specific  needs  of  the  unit.  Many  instructor 
cadre  also  recognize  other  benefits: 


improvements  in  student  motivation,  learning  rates,  and 
performance,  and 

reductions  in  total  training  time  and  costs. 

The  problems  regarding  the  general  application  of  ISD  at  the 
missile  unit  are  more  numerous.  Seven  specific  problems  are 
recognized  by  instructor  cadre: 

constant  turnover  of  military  personnel,  resulting 
in  instructors  who  lack  training  and  experience  in 
performing  ISD  tasks 

lack  of  cooperation  or  support  between  agencies  which 
hinders  the  process 

a  lack  of  sufficient  theoretical  information  to  enable 
long  term  retention  by  students 

higher  headquarters  regulations  which  contradict 
recommendations  of  ISD  models 

the  difficulty  of  developing  concise,  understandable 
objectives 

the  time-consuming  nature  of  developing  training 
using  ISD 

the  large  consumption  of  time  and  money  re-doing  ISD 
tasks  due  to  the  seemingly  endless  series  of  changes 
to  the  weapon  system,  regulations,  and  technical  data. 

The  benefits  and  problems  of  ISD  as  perceived  by  instructor  cadre 
at  the  operational  missile  unit  are  thus  identical  to  those 
encountered  by  training  unit  (i.e.  ATC )  instructors. 
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Missile  unit  instructor  cadre  also  have  definite  opinions 
about  the  problems/benefits  of  the  primary  ISD-related  product  at 
the  missile  unit  -  the  CPOs.  Two  benefits  of  the  CPOs  are 
recognized.  The  greater  benefit,  based  upon  the  percentage  of 
instructor  cadre  who  indicated  agreement  during  the  survey,  is 
the  ability  of  the  CPOs  to  provide  a  solid  foundation  for 
training  at  the  missile  unit.  The  other  benefit  is  an 
improvement  in  communication  between  DOV  and  the  training 
divisions,  probably  because  the  foundation  laid  by  the  CPOs 
applies  to  both  types  of  agencies.  These  conclusions  regarding 
the  benefits  of  the  CPOs  confirm  findings  by  Major  Luethemeier  in 
the  Air  Command  and  Staff  College  research  report  cited  in 
Chapter  II. 

The  findings  of  Major  Leuthemeier,  however,  differ  somewhat 
with  the  conclusions  of  this  thesis  regarding  the  problems  of  the 
CPOs.  Two  aspects  of  the  CPOs  said  to  be  problems  by  Major 
Leuthemeier  are  not  currently  considered  problems  by  the  majority 
of  instructor  cadre  at  the  missile  units.  The  idea  that  CPOs 
convey  a  wrong  signal  by  emphasizing  speed  rather  than  correct 
action  is  rejected  by  two-thirds  of  the  instructor  cadre.  A 
slight  majority  of  instructor  cadre  also  reject  the  notion  that 
CPO  time  standards  are  less  than  accurate.  In  their  minds,  the 
lone  problem  is  the  inhibiting  nature  of  the  CPOs  with  respect  to 
trainer /evaluator  judgement  on  the  MPT. 
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Implications 

The  conclusions  regarding  ISD  models  and  their  problems/ 
benefits  yield  a  reasonably  comprehensive  and  valid  description 
about  the  ISD  process  as  applied  at  the  operational  missile  unit. 
There  are  limitations  to  the  absolute  certainty  of  these 
conclusions,  however.  The  research  design  reduces  certainty  to 
some  extent,  as  discussed  in  Chapter  III.  Better  structured 
response  categories  to  survey  questions  on  the  Five-Step  and  CRI 
Models  would  provide  a  more  conclusive  quantitative  backup  for 
inferences  drawn  through  logic.  A  census  of  missile  unit 
instructors  rather  than  a  small  stratified  sampling  of  instructor 
cadre  also  would  promote  greater  certainty.  The  cadre  surveyed 
by  this  research  effort  generally  have  more  than  two  years  of 
experience  developing  and  conducting  instruction  at  the  missile 
unit,  and  have  accomplished  the  following  ISD-related  training: 
locally-developed  unit  training,  the  SAC  Missile  Instructor 
Course,  and  either  the  Criterion  Referenced  Instruction  Training 
Course  or  the  Wing  Instructional  Systems  Manager  Training  Course. 
It  is  likely  that  cadre  surveyed  for  this  research  effort  have 
more  experience  and  educational  background  developing  and 
conducting  missile  unit  training  than  the  remainder  of  the 
instructor  population.  While  this  experience  and  education  puts 
the  cadre  in  a  position  to  be  knowledgeable  of  the  ISD  process  at 
the  missile  unit,  there  is  a  slight  possibility  that  their  ideas 


are  not  representative  of  the  instructor  population. 


Iff  however,  conclusions  based  on  the  ideas  of  cadre  are 
representative  of  the  instructor  population,  then  there  are  a 
number  of  implications  for  the  missile  units  regarding 
application  of  ISD  in  the  manner  previously  described.  Positive 
and  negative  implications  ca.<  be  identified. 

The  omission  or  modification  of  ISD  steps  should  not 
automatically  imply  negative  connotations  about  application  of 
ISD  within  the  missile  unit.  ISD  models  are  supposed  to  be 
flexible  in  order  to  accommodate  particular  training  needs.  In 
addition,  formal  guidelines  for  developing  training  programs 
using  ISD  are  listed  in  Air  Force  manuals  and  pamphlets,  thus  are 
not  regulatory.  Unfortunately,  however,  a  few  desirable  effects 
on  training  programs  are  lost  when  missile  unit  instructors 
modify  certain  aspects  of  the  models. 

When  missile  units  modify  the  validation  process  to  include 
formal  coordination  and  instructor  tryouts  instead  of  student 
tryouts,  they  do  not  ensure  that  actual  students  can  achieve  the 
objectives  of  training  prior  to  implementation  of  the  training 
program.  Instructors  therefore  begin  a  training  program  with 
less  knowledge  of  the  certainty  of  its  success  than  they  could 
obtain.  While  analysis  of  test  or  evaluation  results  later 
obtains  feedback  on  the  success  of  the  training  program,  it  is 
performed  after  training  has  been  initiated,  when  consequences  of 
inadequate  training  are  more  serious  for  both  instructors  and 


students.  The  lack  of  student  tryouts  is  therefore  an 
unfortunate  circumstance. 

Other  negative  implications  regarding  the  application  of  ISD 
at  the  missile  unit  are  the  lack  of  evaluation  by  instructors  of 
the  media  to  be  used  for  training,  and  the  relative  lack  of 
student  self-pacing.  By  failing  to  comply  more  carefully  with 
the  recommendations  of  the  Five-Step  Model  regarding  media 
evaluation  and  student  self -pacing,  missile  unit  instructors 
potentially  lose  an  element  of  student  involvement  during 
training  sessions.  Students  are  often  less  motivated  to  learn 
when  they  are  instructed  through  media  and  training  methods  which 
are  not  appealing  to  them. 

Each  of  these  negative  implications  result  from  constraints 
within  the  missile  unit  environment.  Constraints  have  been 
previously  identified  in  this  chapter:  the  lack  of  audio-visual 
capability  which  prohibits  the  use  of  potentially  effective 
media;  the  missile  unit  training  regulations  published  by  SAC 
which  specify  minimum  training  periods  for  classroom  training  and 
thus  hinder  student  self-pacing;  and  huge  number  of  deadlines 
which  hinder  accomplishment  of  those  training  tasks  that  are  not 
absolutely  necessary.  The  five  problems  detailed  under  Research 
Question  Two  in  this  chapter  also  contribute  to  the  negative 
implications  listed  above.  Additionally,  the  fact  that 
instructor  cadre  identify  more  problems  than  benefits  regarding 
ISD  application  is  itself  a  negative  implication. 


On  the  positive  side,  the  primary  product  of  ISD  at  the 
missile  unit,  the  CPOs,  is  considered  to  be  a  beneficial  basis 
for  training  and  evaluation  by  93  percent  of  missile  unit 
instructor  cadre.  An  overwhelming  majority  of  the  same  cadre 
regard  the  Five-Step  and  CRI  Models  as  useful  tools.  The  obvious 
implication  from  these  two  positive  notes  is  that  ISD  will  not  be 
abandoned,  regardless  of  the  constraints  or  problems  posed  to  the 
missile  unit. 

Recommendations 

Though  not  within  the  purview  of  the  original  research  goal, 
recommendations  may  enhance  future  application  of  ISD  at  the 
missile  units.  The  basic  recommendation  for  enhancing  the 
application  of  ISD  is  to  remove  as  many  of  the  constraints  and 
problems  identified  in  this  research  report  as  possible. 
Additional  funding  might  be  allotted  for  purchase  of  audio-visual 
equipment.  Personnel  allocations  for  the  training  divisions 
could  be  increased  in  an  effort  to  remove  time  constraints  from 
individuals.  Requirements  for  minimum  classroom  training  periods 
could  be  removed  from  regulations.  These  are  but  three  simple 
examples  of  actions  which  may  eliminate  constraints. 

Headquarters  SAC  would  have  to  be  the  focal  point  for  removal  of 
constraints  due  to  the  nature  of  these  actions.  More  complicated 
actions  might  be  necessary  in  order  to  remove  many  of  the  other 
problems  associated  with  the  general  application  of  ISD.  Because 


many  of  the  problems  identified  in  this  research  report  have  been 
encountered  in  training  units,  consultation  with  ATC  might  offer 
additional  solutions.  Of  course,  any  recommended  solutions  for 
removal  of  constraints  or  problems  must  be  considered  in 
conjunction  with  their  effect  on  MCCM  proficiency.  Costs  of 
enhancing  ISD  application  at  the  missile  unit  may  be  too  large 
unless  they  are  accompanied  by  improvement  in  MCCM  proficiency. 

Another  recommendation  for  enhancing  ISD  application  at  the 
missile  unit  is  to  continue  descriptive  research  on  the  topic. 
This  research  report  could  be  expanded  to  identify  specific 
differences  in  ISD  application  between  training  divisions  or 
weapon  systems  at  the  SAC  missile  units.  Such  expanded  research 
would  promote  greater  understanding  by  instructors  about  the 
development  of  training  programs  within  their  division  and  weapon 
system.  In  order  to  statistically  ascertain  differences  between 
divisions  or  weapon  systems,  the  research  would  have  to  be  based 
upon  a  census  of  the  instructor  population,  or  at  least  a  larger, 
scientifically  drawn  sample  than  the  one  used  in  this  thesis. 

The  increase  in  certainty  promoted  by  a  large  sample  or  census 
would  also  be  an  advantage  over  the  certainty  limitations  within 
this  thesis.  Design  of  any  future  research  efforts  should  also 
seek  improvement  on  other  limitations  previously  described. 
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Final  Summary 


The  recommendations  most  recently  identified  above,  like  the 
positive  and  negative  implications  previously  described  during 
the  chapter,  have  provided  useful  supplemental  information  to 
this  research  effort.  The  overall  goal  of  the  research  effort, 
however,  has  been  to  provide  a  detailed  description  of  ISD  as  it 
relates  to  the  operational  missile  unit.  Explanations  of  the  ISD 
models  applied  at  the  missile  unit  and  their  problems/benefits 
have  been  used  to  fulfill  this  goal.  These  explanations  have 
been  carefully  synthesized  based  upon  a  background  literature 
review,  interviews,  and  a  comp  rehen si ve  survey  of  missile  unit 
instructor  cadre.  Ultimately,  it  is  hoped  that  these 
explanations  have  provided  practical  literature  for  instructors 
and  staff  officers  at  the  missile  unit,  enabling  them  to 
understand,  apply,  and  control  ISD  in  the  development  of  MCCM  job 
training  with  greater  confidence  and  efficiency. 


A  three-day  course  offered  by  a  field  team  at  the  missile 
unit  "to  provide  missile  operations  instructors  the  knowledge, 
skills,  and  interaction  necessary  to  provide  the  best  possible 
instruction  for  the  SAC  missile  operations  force."  Divided  into 
12  subjects:  "self-paced  instruction,  elements  of  the 
professional  instructor,  laws  of  learning,  defense  mechanisms, 
critiquing,  instructional  systems  development  (ISD),  lesson  plan 
construction,  training  aids,  speaking  techniques,  MPT  lab,  stress 
management,  and  lesson  plan  presentation." 

Criterion  Referenced  Instruction  Training  Course 

"Provides  training  in  the  preparation  of  criterion 
referenced  instruction  to  support  instructional  systems 
development  concepts.  Training  includes  analyzing  the  principles 
of  criterion  referenced  instruction;  writing  performance 
objectives;  accomplishing  task/goals  performance  analysis; 
preparing  criterion  tests;  identifying  relevant  course  materials; 
designing  course  control  documents;  and  applying  performance 
management  principles  for  supervisors,  instructors,  and  students. 
This  course  is  primarily  intended  for  developers  of 
instruction."  Course  duration:  three  weeks,  self-paced. 


Mini-CRIT  Course 


"A  combination  of  Mager/Pipe  Criterion  Referenced 
Instruction  (CRI)  and  AF  Instructional  Systems  Development  (ISD) 
methods  of  developing  and  evaluating  instructional  programs. 
Provides  training  for  individuals  involved  in  managing  and 
supervising  the  preparation,  development,  implementation,  and 
evaluation  of  missile  training  programs  at  all  levels  of  command. 
Includes  instruction  in  analyzing  tasks,  determining  education 
and  training  requirements,  developing  objectives  and  tests,  and 
analyzing  goals.  Compares  conventional  or  traditional  training 
to  the  CRI  and  ISD  approach  to  developing  instruction." 
Approximately  one  week  duration,  self-paced. 

Wi ng  Instructional  Systems  Manager  Training  Co urse 

"Provides  training  in  the  basic  concepts  and  applications  of 
learning,  teaching,  and  evaluating.  Includes  study  of  selected 
portions  of  Instructional  Systems  Development  (ISD)  and  Criterion 
Referenced  Instruction  (CRI)  concepts  which  are  used  in 
determination  and  evaluation  of  course  objectives.  In  practice 
training  sessions,  the  students  will  demonstrate  comprehension  of 
these  concepts  and  applications  by  presenting  lessons  which  they 
have  developed  during  course.  Also  includes  training  in  the  use 
of  visual  aids  such  as  sound-slide  and  videotape  as  well  as 
techniques  useful  in  wing  instructional  systems  management." 
Course  duration:  approximately  four  weeks,  self-paced.  For 
persons  assigned  to  ISMD.  (l:vi-viii;  and  3;  4;  and  5) 
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Consent 


Appendix  B:  Instructional  System  Development 
Survey  Interview  Schedule 


Good  morning  (afternoon).  This  is  Captain  Guy  Fritchman  at 
Wright-Patterson  AFB .  I 'm  a  former  EWO  instructor  and  Wing 
Instructional  System  Manager  at  McConnell  AFB.  Currently  I 'm 
studying  for  my  Master's  degree  at  AFIT.  As  part  of  my  studies, 

I 'm  conducting  interviews  to  gather  information  on  ISD  as  it 
applies  to  operational  missile  units.  I'm  trying  to  interview 
one  person  from  each  training  division  at  each  unit.  Ultimately, 
I  hope  this  research  will  provide  a  practical  description  for  new 
instructors  and  staff  members  about  ISD  as  it  applies  to  the 
missile  unit. 

(When  talking  to  Chief,  ISMD)  Would  you  be  willing  to 
participate? 

(When  talking  to  other  persons)  (Name  of  Chief,  ISMD 
at  that  unit)  said  you  would  be  a  good 
person  to  interview.  Would  you  be  willing  to 
participate? 

Appointment 

The  interview  will  take  approximately  40-50  minutes  of  your 
time.  Would  now  be  a  good  time,  or  can  I  make  an  appointment  to 
call  you  back  later? 
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(If  interview  done  by  appointment,  call  back  at  scheduled 
time  and  reintroduce  myself  and  the  purpose  of  the 
interview,  then  continue  with  this  interview  schedule.) 

Survey  Introduction 

Structure  .  The  interview  I 'm  about  to  conduct  has  five 
basic  parts.  Part  I  requests  general  background  information  from 
you.  A  Five-Step  ISD  Model  is  described  in  Part  II  while 
soliciting  your  experience,  knowledge,  and  opinions  of  this 
model.  Similar  information  is  solicited  in  Part  III  about  a 
model  by  Dr  Robert  F.  Mager.  Your  opinion  of  the  problems  and 
benefits  of  the  Comprehensive  Performance  Objectives  is  sought  in 
Part  IV.  Finally,  Part  V  examines  your  opinion  of  ISD  in 
general . 

Addi tional  Items .  A  few  other  brief  items  I  want  you  to 
be  aware  of  before  we  get  started: 

This  survey  is  being  accomplished  primarily  for  research 
purposes,  to  fulfill  the  requirements  for  my  degree  at 
AFIT.  It  is  not  commissioned  by  HQ  SAC,  although  it  has 
been  coordinated  through  DOMM  (Major  Maggio). 

Your  name  will  not  be  used  when  the  results  of  this 
survey  are  written. 
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And  finally,  many  of  the  questions  I'm  about  to  ask  are 
structured  and  require  you  to  choose  from  the  answers 
I'll  provide.  Feel  free,  however,  to  provide  any 
additional  comments  you  desire. 

That  concludes  my  introduction.  Do  you  have  any  questions 
or  comments  before  we  get  started? 

Part  I:  Background  Information 

1.  How  much  experience  have  you  had  developing  or  conducting 
instruction  at  the  missile  unit? 

a.  less  than  6  months 

b.  at  least  6  months  but  less  than  1  year 

c.  at  least  1  year  but  less  than  1  1/2  years 

d.  at  least  1  1/2  years  but  less  than  2  years 

e .  2  years  or  more 

2.  What  instructor  training  have  you  accomplished? 

a.  Locally  developed  unit  training 

b.  SAC  Missile  Instructor  course 

c.  Criterion  Referenced  Instruction  Training  (CRIT)  course 

d.  Mini-CRIT  course 

e.  Wing  Instructional  Systems  Manager  Training  course 

f.  Academic  Instructor  School 

g .  Other  : _ 


Five -Step  ISP  Model 


One  model  accepted  within  SAC  is  a  five-step  process  as 
prescribed  by  AFM  50-2.  To  develop  a  common  basis  for  discussing 
the  use  of  this  model  within  your  unit,  I  would  like  to  briefly 
describe  my  understanding  of  each  step,  pausing  between  each 
description  to  ask  you  a  few  questions. 

Step  1  of  the  model  recommends  that  instructors  begin  the 
process  of  developing  a  job  training  program  with  "task 
analysis."  Here  instructors  should  survey,  question,  or  observe 
workers,  and  analyze  regulations,  checklists,  and  technical  data 
about  the  job  in  order  to  determine  its  human  performance 
requirements.  Through  such  analysis,  the  instructor  compiles  a 
list  of  the  knowledge,  skills,  behaviors,  duties,  and  tasks 
required  by  workers  to  perform  their  jobs  in  a  satisfactory 
manner . 

3.  Does  your  unit  accomplish  task  analysis  similar  to  Step  1  in 
order  to  produce  a  list  of  the  knowledge,  skills,  behaviors, 
duties,  and  tasks  required  of  missile  combat  crew  members? 

a .  Often 

b.  Sometimes 

c.  Never 


d .  Don' t  Know 


4.  Are  you  aware  of  any  product  within  your  unit  or  SAC 
which  can  be  regarded  as  the  result  of  Step  1?  If  so  please 
specify? 


The  goal  of  Step  2  of  this  model  is  to  determine  the  exact 
instruction  needed  to  qualify  workers  to  do  the  job.  This 
determination  is  based  upon  the  analysis  in  Step  1  and  upon 
considerations  such  as:  tasks  which  potential  students 
cannot  already  accomplish;  consequence  of  inadequate 
performance;  how  fast  the  person  must  react  to  status 
change;  frequency  of  performance;  availability  of 
instructors,  equipment,  and  facilities;  long  lead  time 
resources;  and  time  for  training.  The  tasks,  knowledges, 
and  proficiencies  identified  for  training  are  compiled  into 
instructional  standards,  a  specialized  publication  used  to 
standardize  and  control  training. 

5.  Does  your  unit  accomplish  the  tasks  listed  in  Step  2  in  order 
to  produce  a  set  of  instructional  standards? 

a.  Often 

b.  Sometimes 

c.  Never 

d  .  Don ' t  Know 


6.  Are  you  aware  of  any  product  within  your  unit  or  SAC 
which  can  be  regarded  the  result  of  Step  2?  If  so,  please 
spec  if  y . 
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each  of  the 


At  Step  3,  instructors  develop  objectives  for 
instructional  standards  of  the  previous  step.  Objectives 
are  specific  statements  labeling  each  behavior  required  of  a 
student,  the  conditions  under  which  each  behavior  must  be 
accompli  shed ,  and  the  level  of  excellence  or  standards  for 
accomplishment.  Criterion  tests,  also  developed  during  this 
step,  serve  as  the  basis  for  measuring  the  students' 
performance,  as  specified  by  the  objectives. 

7.  Does  your  unit  use  written  objectives  as  the  basis  for 
training? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

8.  Written  tests  at  your  unit  are  based  upon  specific 
objectives . 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don ' t  Know 
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9.  MPT  scripts  produced  by  your  unit  are  based  upon  specific 
objectives . 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don ' t  Know 

10.  Please  cite  examples  of  criterion  tests  used  by  your 
unit  which  require  actual  demonstration  of  the  student' s 
performance . 

During  the  fourth  step  of  this  model,  the  instructor  plans, 
develops,  and  validates  the  actual  instruction  which  will  be 
presented  to  the  student.  The  model  recommends  that 
instructors  sequence  objectives  in  a  manner  which  produces 
required  learning  in  the  shortest  time.  It  also  encourages 
instructors  to  select  methods  of  instruction  whereby 
students  can  participate  or  individually  pace  themselves 
toward  achieving  objectives.  Furthermore,  careful  selection 
of  media  is  encouraged,  so  that  information  about  the 
objective  is  provided  in  a  manner  interesting  to  the 
student,  but  at  the  same  time  does  not  become  a  burden  on 
the  instructional  system.  By  media,  I  am  referring  to 
instructors,  textbooks,  programmed  texts,  simulators, 
slides,  videotapes,  etc.  Other  final  courseware,  such  as 


lesson  plans,  study  guides,  and  Plans  of  Instruction  are 
produced  at  Step  4.  When  final  courseware  has  been 
developed,  validation  of  the  training  program  is 
recommended,  using  a  small  sample  of  students  to  determine 
if  they  can  achieve  the  objectives  through  the  planned 
instruction . 

11.  Does  your  unit  make  an  effort  to  sequence  objectives 
and  instruction  in  a  manner  which  is  felt  to  best  promote 
student  learning? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

12.  At  your  unit: 

a.  Student  self -pacing  methods  are  used  more  than 
lecture 

b.  Student  self -pacing  and  lecture  are  used  equally 

c.  Lecture  is  used  more  than  student  self -pacing 

d .  Don' t  Know 

13.  Please  cite  examples  of  methods  in  which  student 
participation  is  encouraged  within  the  training  programs  of  your 
unit . 
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14.  Does  your  unit  evaluate  the  advantages  and  disadvantages  of 
various  media  when  developing  instruction? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don ' t  Know 

15.  Does  your  unit  perform  a  tryout  on  a  small  group  of  students 
in  order  to  validate  instruction? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

16.  Please  cite  any  other  method  you  would  consider  as  a 
means  of  validating  instruction. 

The  fifth  step  of  the  model  involves  conduct  of  training  (in 
accordance  with  the  plan  developed  in  Step  4)  and  evaluation 
of  its  adequacy.  There  are  two  types  of  evaluation: 
internal  and  external.  Internal  evaluation  is  similar  to 
validation  —  it  allows  the  evaluator  to  judge  whether  the 
training  system  still  enables  the  student  to  achieve  the 
objectives.  External  evaluation  examines  whether  the  tasks 
required  of  the  job  for  which  the  training  system  exists  are 


still  the  same  as  when  the  original  task  analysis  was 
performed . 

17.  How  does  your  unit  conduct  internal  evaluation  on  its 
training  programs? 

18 .  Which  of  the  following  would  you  consider  external 
evaluation?  (More  than  one  response  may  be  appropriate) 

a.  Standardization/Training  Conferences 

b.  CPO  conferences 

c.  EWO  conferences 

d.  Higher  Headquarters  evaluations 

e.  Other:  _ 

Part  III :  Criterion  Referenced  Instruction  Model 

Another  model  accepted  by  SAC  is  one  developed  by 
Dr.  Robert  F.  Mager.  It  is  similar  to  the  Five-Step  Model; 
however,  it  provides  additional  contributions  to  developing 
training  programs.  One  contribution  is  goal  analysis. 
Unlike  task  analysis,  goal  analysis  identifies  cognitive 
areas  of  potential  training  such  as  attitudes  rather  than 
tasks  or  skills  required  by  persons.  It  could  be  desired, 
for  example,  that  individuals  show  pride,  finesse,  or 
enthusiasm  in  their  work.  It  is  the  task  of  the  instructor 
to  identify  such  goals,  then  determine  specific  student 


performances  which,  if  accomplished,  indicate  achievement  of 
the  goals. 

19.  Does  your  unit  perform  goal  analysis  when  developing 
training  programs? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

Mager  furthermore  recommends  that  most  instruction  be 
presented  in  the  following  manner:  first,  students  are 
given  a  set  of  objectives  to  achieve.  Necessary  information 
about  each  specific  objective  is  presented  in  the  form  of 
explanations,  examples,  or  pictures.  Other  sources  of 
information  are  usually  also  identified  to  students.  Next, 
the  students  are  given  practice  in  achieving  each  objective, 
and  they  are  questioned  about  each  objective  as  they 
practice,  enabling  each  of  them  to  self-measure  progress. 
Correct  answers  indicate  that  a  particular  student  is 
capable  of  achieving  the  objective.  On  the  other  hand, 
incorrect  answers  necessitate  additional  information  and 
practice,  which  should  be  provided.  Finally,  as  students 
complete  practice  at  their  own  pace,  each  is  presented  with 
a  criterion  test  as  a  final  measure  of  ability  in  achieving 
the  objective. 


20.  Does  your  unit  present  instruction  in  the  manner 
recommended  by  Mager? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

Part  IV:  Comprehensive  Performance  Objectives  ( CPOs ) 

For  this  part  of  the  interview,  I  seek  your  opinion  on 
various  aspects  of  the  CPOs.  Here  I  wish  to  know  the 
strength  of  your  opinion;  therefore,  I  ask  that  you  respond 
with  one  of  the  following  answers  for  each  question: 
Strongly  agree 
Agree 
Uncertain 
Disagree 

Strongly  disagree 

I'll  repeat  these  possible  choices  after  each  question. 

21.  The  CPOs  provide  a  beneficial  basis  for  training  and 
evaluation . 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


m  ■  *  »  • 
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22.  Improved  communication  between  DOV  and  the  training 
divisions  is  one  benefit  of  the  CPOs. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 

23.  CPOs  have  accurate  and  appropriate  time  standards. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 

24.  CPOs  inhibit  trai ner /evaluator  judgement  on  the  MPT. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


2  5.  CPOs  convey  the  wrong  signal  by  emphasizing  speed 
rather  than  correct  action. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 

26.  Are  there  any  other  benefits  or  problems  with  the  CPOs 
that  you  would  care  to  discuss? 

Part  V:  Gene ral  Opi nion  of  ISP 

In  this  final  part  of  the  interview,  I  will  read  to  you 
a  series  of  statements  reflecting  benefits  and  problems  said 
to  be  caused  by  using  ISD  at  training  units  (i.e  ATC  units). 
Please  indicate  after  each  statement  whether  you  believe  it 
to  be  applicable  at  the  operational  missile  unit.  A  simple 
yes/no  answer  will  suffice,  unless  you  have  additional 
comments . 
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27.  Which  of  the  following  statements  reflect  your  opinion 
of  the  benefits  of  using  ISD  at  the  operational  missile 
unit . 

a.  Student  performance  has  increased. 

b.  Student  learning  rates  have  improved. 

c.  Student  motivation  has  improved. 

d.  Total  training  time  is  reduced. 

e.  Costs  are  reduced. 

f.  ISD  provides  definite,  yet  flexible  guidelines  for 
developing  and  conducting  instruction. 

g.  I  don't  see  any  benefits  in  using  ISD  at  my  unit. 


h.  Other 


28.  Which  of  the  following  reflect  your  opinion  of  the 
problems  caused  by  using  ISD  at  the  operational  missile 
unit? 


a.  It  is  extremely  time-consuming  to  develop  training 
using  ISD. 

b.  Developing  concise,  understandable  objectives  is  a 
difficult  task. 

c.  Lack  of  cooperation  or  support  between  agencies 
hinders  the  process. 

d.  Constant  turnover  of  military  personnel  results  in 
instructors  who  lack  training  and  experience  in 
performing  ISD  tasks. 

e.  Higher  Headquarters  regulations  and  directives 
often  interfere  with  the  process. 

f.  The  seemingly  endless  series  of  changes  to  the 
weapon  system,  regulations,  and  technical  data 
hinders  the  ISD  process.  Much  time  and  money  is 
consumed  re-doing  tasks. 

g.  ISD-developed  training  programs  lack  sufficient 
theoretical  information  to  enable  long-term 
retention  (in  other  words,  students  can  absorb 
material  quickly,  pass  a  test,  then  forget  it 
quickly ) . 

h.  I  don't  see  any  problems  using  ISD  at  my  unit. 


i .  Other 


29.  This  concludes  the  structured  part  of  my  survey;  however, 
would  appreciate  any  other  comments  regarding  this  survey,  ISD 
general,  or  any  specific  aspects  of  ISD. 

(When  interviewing  Chief,  ISMD,  ask  for  names  of  potential 
participants  from  DOTI ,  D022,  and  D09.) 

Thank  you  for  participating.  Your  information  has  been  very 
helpful.  I  wish  you  best  of  luck. 


Appendix  C:  Instructional  System  De velopment 
( ISP )  Questionnai re 


INTRODUCTION:  ISP  models  provide  f lexi ble  guidelines  for 

developing,  conducting,  and  evaluating  instruction.  Though  some 
ISD  models  provide  a  step-by-step  procedure,  application  of  the 
steps  and  tasks  within  the  steps  will  differ  depending  upon  the 
situation.  The  application  of  ISD  at  your  missile  unit,  for 
example,  is  probably  different  than  at  training  units  within  the 
Air  Training  Command.  This  survey  is  used  to  gather  data  on  ISD 
as  it  applies  to  the  operational  missile  unit.  Results  of  the 
survey  will  be  compiled  into  the  research  report  of  a  former 
missile  weapon  system  and  EWO  instructor  now  at  the  Air  Force 
Institute  of  Technology.  Ultimately  it  is  hoped  that  the 
research  report  will  provide  a  practical  description  for  new 
instructors  and  staff  members  about  ISD  as  it  applies  to  the 
missile  unit. 

SURVEY  STRUCTURE:  This  survey  has  five  basic  parts.  Part  I 
requests  general  background  information  from  you.  A  Five-Step 
ISD  Model  is  described  in  Part  II  while  soliciting  your 
experience,  knowledge,  and  opinions  of  this  model.  Similar 
information  is  solicited  in  Part  ill  about  a  Criterion-Referenced 
Instruction  Model.  Your  opinion  of  the  benefits  and  problems  of 
the  Comprehensive  Performance  Objectives  is  sought  in  Part  IV. 
Finally,  Part  V  examines  your  opinion  of  ISD  in  general. 
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DIRECTIONS  FOR  COMPLETING  THIS  SURVEY:  Many  questions  in  this 
survey  provide  a  list  of  possible  answers.  For  these  questions, 
please  circle  the  answer  which  best  indicates  your  opinion  or 
choice.  Unless  specified  in  the  question,  please  circle  only  one 
answer.  Other  questions  are  open-ended  questions;  please  provide 
comments  as  appropriate  to  these  questions  in  the  space  provided. 
You  are  welcome  to  provide  additional  comments  at  the  end  of  this 
survey  or  beside  any  question. 

Part  I:  Gene ral  Information 
1.  To  which  unit  are  you  assigned? 

a.  44  SMW 

b.  90  SMW 

c .  91  SMW 

d .  308  SMW 

e.  321  SMW 

f  .  341  SMW 

g.  351  SMW 

h.  38  1  SMW 


2.  To  which  division  are  you  assigned? 

a.  DOT I 

b.  DO 2 2 

c.  DO  9 
d  .  DO  5 
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3.  How  much  experience  have  you  had  developing,  conducting,  or 
evaluating  instruction  at  the  missile  unit? 


a.  Less  than  6  months 

b.  At  least  6  months  but  less  than  1  year 

c.  At  least  1  year  but  less  than  1  1/2  years 

d.  At  least  1  1/2  years  but  less  than  2  years 

e.  2  years  or  more 

4.  What  instructor  training  have  you  accomplished?  (Circle  all 
applicable  choices.) 

a.  Locally  developed  unit  training 

b.  SAC  Missile  Instructor  Course 

c.  Criterion  Referenced  Instruction  Training  (CRIT) 

Course 

d.  Mini-CRIT  Course 

e.  Wing  Instructional  Systems  Manager  Training  Course 

f.  Academic  Instructor  School 

g .  Other  :  _ 


Part  II :  Five-Step  ISP  Model 

One  ISD  model  accepted  within  SAC  is  a  five-step  process. 
Step  1  of  the  model  recommends  that  instructors  begin  the  process 
of  developing  a  job  training  program  with  "task  analysis."  Here 


instructors  should  survey,  question,  or  observe  workers,  and 


analyze  regulations,  checklists,  and  technical  data  about  the  job 
in  order  to  determine  its  human  performance  requirements. 

Through  such  analysis,  the  instructor  compiles  a  list  of 
knowledge,  skills,  behaviors,  duties,  and  tasks  required  by 
workers  to  perform  their  jobs  in  a  satisfactory  manner. 

5.  Does  your  unit  accomplish  task  analyses  similar  to  Step  1  in 
order  to  produce  a  list  of  knowledge,  skills,  behaviors,  duties, 
and  tasks  required  of  missile  combat  crew  members? 

a.  Often 

b.  Sometimes 

c.  Never 

d  .  Don ' t  Know 

6.  Are  you  aware  of  any  product  within  your  unit  or  SAC  which 
can  be  regarded  as  the  result  of  Step  1?  If  so,  please  specify. 


The  goal  of  Step  2  of  this  model  is  to  determine  the  exact 
instruction  needed  to  qualify  workers  to  do  the  job.  This 
determination  is  based  upon  the  analysis  in  Step  1  and  upon 
considerations  such  as:  tasks  which  potential  students  cannot 
already  accomplish;  consequence  of  inadequate  performance;  how 
fast  the  person  must  react  to  status  change;  frequency  of 
performance;  availability  of  instructors,  equipment,  and 


facilities;  long  lead  time  resources;  and  time  for  training.  The 
tasks,  knowledges,  and  proficiencies  identified  for  training  are 
compiled  into  instructional  standards,  a  specialized  publication 
used  to  standardize  and  control  training. 

7.  Does  your  unit  accomplish  the  tasks  listed  in  Step  2  in 
order  to  produce  a  set  of  instructional  standards? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


8.  Are  you  aware  of  any  product  within  your  unit  or  SAC  which 
can  be  regarded  the  result  of  Step  2?  If  so,  please  specify. 


At  Step  3,  instructors  develop  objectives  for  each  of  the 
instructional  standards  of  the  previous  step.  Objectives  are 
specific  statements  labeling  each  behavior  required  of  a  student 
the  conditions  under  which  each  behavior  must  be  accomplished, 
and  the  level  of  excellence  or  standards  for  accomplishment. 
Criterion  tests,  also  developed  during  this  step,  serve  as  the 
basis  for  measuring  the  student's  performance,  as  specified  by 
the  objectives. 


9.  Does  your  unit  use  written  objectives  as  the  basis  for 
training? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

10.  Written  tests  at  your  unit  are  based  upon  specific 
objectives . 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

11.  MPT  scripts  produced  by  your  unit  are  based  upon  specific 
objectives . 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

12.  Please  cite  examples  of  criterion  tests  used  by  your  unit 
which  require  actual  demonstration  of  the  student's  performance. 


During  the  fourth  step  of  this  model,  the  instructor  plans, 
develops,  and  validates  the  actual  instruction  which  will  be 
presented  to  the  student.  The  model  recommends  that  instructors 
sequence  objectives  in  a  manner  which  produces  required  learning 
in  the  shortest  time.  It  also  encourages  instructors  to  select 
methods  of  instruction  whereby  students  can  participate  or 
individually  pace  themselves  toward  achieving  objectives. 
Furthermore,  careful  selection  of  media  (instructor,  textbook, 
programmed  text,  simulator,  slides,  videotape,  etc.)  is 
encouraged,  so  that  .information  about  the  objective  is  provided 
in  a  manner  interesting  to  the  student,  but  at  the  same  time  doe 
not  become  a  burden  on  the  instructional  system.  Other  final 
courseware,  such  as  lesson  plans,  study  guides,  and  Plans  of 
Instruction  are  produced  at  Step  4.  When  final  courseware  has 
been  developed,  validation  of  the  training  program  is 
recommended,  using  a  small  sample  of  students  to  determine  if 
they  can  achieve  the  objectives  through  the  planned  instruction. 

13.  Does  your  unit  make  an  effort  to  sequence  objectives  and 
instruction  in  a  manner  which  is  felt  to  best  promote  student 
learning? 

a.  Often 

b.  Sometimes 

c.  Never 


d .  Don' t  Know 


14  .  At  your  unit : 

a.  Student  self -pacing  methods  are  used  more  than  lecture. 

b.  Student  self-pacing  and  lecture  are  used  equally. 

c.  Lecture  is  used  more  than  student  self-pacing. 

d .  Don' t  Know 

15.  Please  cite  examples  of  methods  in  which  student 
participation  is  encouraged  within  the  training  programs  of  your 
unit . 


16.  Does  your  unit  evaluate  the  advantages  and  disadvantages  of 
various  media  when  developing  instruction? 


a.  Often 


b.  Sometimes 


c.  Never 


d .  Don' t  Know 


17.  Does  your  unit  perform  a  tryout  on  a  small  group  of  students 
in  order  to  validate  instruction? 


a.  Often 


b.  Sometimes 


c.  Never 


d .  Don' t  Know 


The  fifth  step  of  the  model  involves  conduct  of  training  (in 
accordance  with  the  plan  developed  in  Step  4)  and  evaluation  of 
its  adequacy.  There  are  two  types  of  evaluation:  internal  and 
external.  Internal  evaluation  is  similar  to  validation  -  it 
allows  the  evaluator  to  judge  whether  the  training  system  still 
enables  the  student  to  achieve  the  objectives.  External 
evaluation  examines  whether  the  tasks  required  of  the  job  for 
which  the  training  system  exists  are  still  the  same  as  when  the 
original  task  analysis  was  performed. 

19.  How  does  your  unit  conduct  internal  evaluation  on  its 
training  programs? 


20.  Please  circle  those  functions  which  you  would  consider 
external  evaluation  (more  than  one  may  be  appropriate). 


a.  Standardization/Training  Conferences 

b.  CPO  Conferences 

c.  EWO  Conferences 

d.  Higher  Headquarters  Evaluations 

e.  Other: 


Part  III:  Criterion  Referenced  Instruction  Model 


Another  model  accepted  by  SAC  is  one  developed  by 
Dr.  Robert  F.  Mager.  It  is  similar  to  the  Five-St^p  Model; 
however,  it  provides  additional  contributions  to  developing 
training  programs.  One  contribution  is  goal  analysis.  Unlike 
task  analysis,  goal  analysis  identifies  cognitive  areas  of 
potential  training,  such  as  attitudes  rather  than  tasks  or  skills 
required  by  persons.  It  could  be  desired,  for  example,  that 
individuals  show  pride,  finesse,  or  enthusiasm  in  their  work.  It 
is  the  task  of  the  instructor  to  identify  such  goals,  then 
determine  specific  student  performances  which,  if  accomplished, 
indicate  achievement  of  the  goals. 


21.  Does  your  unit  perform  goal  analysis  when  developing 
training  programs? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 

Mager  furthermore  recommends  that  most  instruction  be  presented 
in  the  following  manner:  first,  students  are  given  a  set  of 
objectives  to  achieve.  Necessary  information  about  each  specific 
objective  is  presented  in  the  form  of  explanations,  examples,  or 
pictures.  Other  sources  of  information  are  usually  also 
identified  to  students.  Next,  the  students  are  given  practice  in 
achieving  each  objective,  and  they  are  questioned  about  each 
objective  as  they  practice,  enabling  each  of  them  to  self-measure 
progress.  Correct  answers  indicate  that  a  particular  student  is 
capable  of  achieving  the  objective.  On  the  other  hand,  incorrect 
answers  necessitate  additional  information  and  practice,  which 
should  be  provided.  Finally,  as  students  complete  practice  at 
their  own  pace,  each  is  presented  with  a  criterion  test  as  a 
final  measure  of  ability  in  achieving  the  objective. 
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22.  Does  your  unit  present  instruction  in  the  manner  recommended 
by  Mager? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don ' t  Know 


Part  IV:  Your  Opinion  of  the  Comp  re  he  ns  i  ve  Pe  rformance 
Objectives  (CPOs ) 

23.  The  CPOs  provide  a  beneficial  basis  for  training  and 
evaluation . 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


24.  Improved  communication  between  DOV  and  the  training 
divisions  is  one  benefit  of  the  CPOs. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 
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25.  CPOs  have  accurate  and  appropriate  time  standards. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 

26.  CPOs  inhibit  trainer /evaluator  judgement  on  the  MPT. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 

27.  CPOs  convey  the  wrong  signal  by  emphasizing  speed  rather 
than  correct  action. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


28.  Please  list  any  other  benefits/problems  you  see  with  the 
CPOs. 

Benefits:  _ 


Problems : 


Part  V:  Yo ur  Opinion  of  ISP  i n  General 

29.  Please  circle  all  the  responses  which  reflect  your  opinion 
of  the  benefits  of  using  ISD  at  your  unit. 

a.  Student  performance  has  increased. 

b.  Student  learning  rates  have  improved. 

c.  Student  motivation  has  improved. 

d.  Total  training  time  is  reduced. 

e.  Costs  are  reduced 

f.  ISD  provides  definite,  yet  flexible  guidelines  for 
developing  and  conducting  instruction. 

g.  I  don't  see  any  benefits  in  using  ISD  at  my  unit. 


h .  Other 


30.  Please  circle  all  the  responses  which  reflect  your  opinion 
of  the  problems  caused  by  using  ISD  at  your  unit. 

a.  It  is  extremely  time-consuming  to  develop  training  using 
ISD. 

b.  Developing  concise,  understandable  objectives  is  a 
difficult  task  for  many  persons. 

c.  Lack  of  cooperation  or  support  between  agencies  hinders 
the  process. 

d.  Constant  turnover  of  military  personnel  results  in 
instructors  who  lack  training  and  experience  in 
performing  ISD  tasks. 

e.  Higher  headquarters  regulations  and  directives 
often  interfere  with  the  process. 

f.  The  seemingly  endless  series  of  changes  to  the  weapon 
system,  regulations,  and  technical  data  hinders  the  ISD 
process.  Much  time  and  money  is  consumed  re-doing 
tasks . 

g.  ISD-developed  training  programs  lack  sufficient 
theoretical  information  to  enable  long-term  retention 
(in  other  words,  students  can  absorb  material  quickly, 
pass  a  test,  then  forget  it  quickly). 

h.  I  don't  see  any  problems  using  ISD  at  my  unit. 


i .  Other : 


31.  Any  other  comments  regarding  ISD  are  appreciated. 


Thank  you  for  your  time  and  comments.  Please  return  this  survey 
to : 

Captain  Guy  Fritchman 
AFIT/LS 

Building  641,  Area  B 
Wright-Patterson  AFB  OH  4  5433 


1.  How  much  experience  have  you  had  developing  or  conducting 
instruction  at  the  missile  unit? 


i 


*  , 
£ 


a . 

b. 

c . 

d. 

e . 


less  than  6  months 

at  least  6  months  but  less  than  1  year 
at  least  1  year  but  less  than  1  1/2  years 
at  least  1  1/2  years  but  less  than  2  years 
2  years  or  more 


Unit 

Respondent 

Answer 

44  SMW 

ISMD 

e 

Ellsworth 

DOTI 

e 

DO  2  2 

e 

DO  9 

d 

90  SMW 

ISMD 

e 

F.E.  Warren 

DOTI 

e 

DO  2  2 

e 

DO  9 

b 

91  SMW 

ISMD 

e 

Minot 

DOTI 

b 

D022 

e 

D09 

c 

308  SMW 

ISMD 

d 

Little  Rock 

DOTI 

e 

DO  2  2 

d 

321  SMW 

ISMD 

c 

Grand  Forks 

DOTI 

e 

D022 

e 

DO  9 

c 

341  SMW 

ISMD 

e 

Malms  trom 

DOTI 

e 

DO  2  2 

e 

DO  9 

a 

3  51  SMW 

ISMD 

e 

Whiteman 

DOTI 

e 

DO  2  2 

e 

DO  9 

e 
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Unit  Respondent  Answer 


381  SMW  ISMD  e 

McConnell  DOTI  c 


2 


What  instructor  training  have  you  accomplished? 


a.  Locally  developed  unit  training 

b.  SAC  Missile  Instructor  course 

c.  Criterion  Referenced  Instruction  Training  (CRIT)  course 

d.  Mini-CRIT  course 

e.  Wing  Instructional  Systems  Manager  Training  course 

f.  Academic  Instructor  School 

g.  Other: _ 


Unit 

Respondent 

Answer 

Comments 

44  SMW 

ISMD 

a,b,d,e 

plus  Masters  and  nearly 

Ellsworth 

DOT  I 

a,  b,d 

PhD  in  Education 

D022 

a 

DO  9 

a  ,b,c 

90  SMW 

ISMD 

F.E.  Warren 

DOT  I 

a,d 

DO  2  2 

a  ,b  ,c 

ECI  Course:  Principles 
and  Techniques  of 
Instruction 

DO  9 

a,f 

91  SMW 

ISMD 

a,b  ,e 

Minot 

DOT  I 

a,b,g 

ECI  Course:  Principles 
and  Techniques  of 
Instruction 

DO  2  2 

a  ,b  ,c 

DO  9 

b,c 

308  SMW 

ISMD 

a,b,e 

ATC  ISD  course  at  Carswell 

Little  Rock 

DOT  I 

a,b,c,e,f 

DO  2  2 

a,f 

321  SMW 

ISMD 

b,c,e 

Grand  Forks 

DOT  I 

a,b  ,d 

DO  2  2 

a ,b ,d  ,f  ,g 

OJT  Supervisor  Training; 
OJT  Program  Development 
Training 

DO  9 

a 
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Unit 


Respondent  Answer 


Comments 


341  SMW 

a,  b,g 

ECI  Course:  Principles 

Malms trom 

and  Techniques  of 
Instruction;  various  ATC 

courses 

DOTI 

a, b,f  ,g 

College  courses 

DO  2  2 

a,b,g 

Technical  Instruction 
School  at  Sheppard 

DO  9 

a ,  g 

ECI  Course:  Principles 
and  Techniques  of 
Instruction 

351  SMW 

ISMD 

a,b,c,e 

Whiteman 

DOTI 

a,  b 

DO  2  2 

a ,  g 

ISD  course  at  Sheppard 

DO  9 

a 

381  SMW 

ISMD 

a,  b,e 

McConnell 

DOTI 

3.  f  b  f  c 

ECI  Course:  Principles 

and  Techniques 
of  Instruction 
Others:  7  ATC  courses  in 

I SO ,  Tests  &  Measurement, 


Developing  Criterion 
Objectives,  etc. 


DO  2  2 


a, b,f ,g 


3.  Does  your  unit  accomplish  task  analysis  similar  to  Step  1  in 
order  to  produce  a  list  of  the  knowledge,  skills,  behaviors, 
duties,  and  tasks  required  of  missile  combat  crew  members? 


Often 

Sometimes 

Never 

Don' t  Know 


Respondent 


An  swe  r 


44SMW 

Ellsworth 

I  SI 
DO' 
DO: 

nn 

90  SMW 

P.E.  Warren 

uu 

I  si 
DO' 
DO 
nn 

91  SMW 

Minot 

uu 

I  si 
DO' 
DO 
DO 

308  SMW 
Little  Rock 

I  SI 
DO’ 
DO 

321  SMW 
Grand  Forks 

I  SI 
DO' 
DO 
nn 

341  SMW 
Malms trom 

uu 

I  SI 
DO' 
DO 
nn 

3  51  SMW 

uu 

I  SI 

Whiteman 

DO' 

DO 

nn 

38 1  SMW 
McConnell 

uu 

IS! 

DO* 

Comments 


Seldom 


Seldom 


Seldom 


4.  Are  you  aware  of  any  product  within  your  unit  or  SAC  which 
can  be  regarded  as  the  result  of  Step  1?  If  so  please  specify? 


Unit 


Respondent 


Comments 


44  SMW 

ISMD 

Task  analysis  performed  when  new 

Ellsworth 

61 6A  introduced;  also  performed 
when  MCC  maintenance  on  LCCs 

introduced 

DOTI 

CPOs 

DO  2  2 

CPOs  and  a  unit  Weapon  System 
Programmed  Text 

DO  9 

JPRs 

90  SMW 

ISMD 

CPOs 

F.E.  Warren 

DOTI 

LCC  maintenance  had  a  task 
analysis  performed;  CPOs 

DO  2  2 

CPOs 

DO  9 

CPOs;  JPRs;  SAC  developing  a 
task  analysis 

91  SMW 

ISMD 

Task  analysis  done  on  LCC 

Minot 

DOTI 

maintenance  training  at  unit 

CPOs  and  JPRs 

DO  2  2 

CPOs 

DO  9 

JPRS 

308  SMW 

ISMD 

CPOs  and  JPRs;  at  unit,  have 

Little  Rock 

DOTI 

done  analysis  on  transition 
training  program 

CPOs  and  JPRs 

DO  2  2 

- 

321  SMW 

ISMD 

Task  analysis  performed  on  LCC 

Grand  Forks 

DOTI 

maintenance  procedures;  CPOs 

CPOs 

DO  2  2 

CPOs  and  nothing  else.  With 
respect  to  EWO,  policy  and 
procedures  set  at  HQ  SAC  and  JCS 
levels  with  no  unit  input 

DO  9 

JPRs 

341  SMW 

ISMD 

CPOs  and  JPRs;  EWO  and  codes  do 

Malmstrom 

task  analyses;  new  crew 
maintenance  procedures 

Unit 


Respondent  Comments 


3  51  SMW 
Whiteman 


38  1  SMW 
McConnell 


ISMD  This  is  mostly  a  human  resources 

step  done  by  other  agencies 
(i.e.  SAC);  only  done  at  unit 


for  new  requirements  (CPOs) 

DOTI 

CPOs  and  JPRs 

0022 

CPOs;  task  analysis  for  new 
procedures  in  TOs 

DO  9 

JPRs 

ISMD 

Transition  training  to  MCCC 
based  on  task  analysis 

DOTI 

CPOs 

D022 

Coverage  rates  of  specific 
subjects  listed  in  SACR  50-21 
are  based  on  frequency  of  task 
performance  and  complexity  of 
task 

5.  Does  your  unit  accomplish  the  tasks  listed  in  Step  2  in 
order  to  produce  a  set  of  instructional  standards? 


a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


Unit  Respondent  Answer  Comment 


44  SMW 

ISMD 

b 

Ellsworth 

a 

D022 

c 

DO  9 

c 

90  SMW 

ISMD 

b 

F.E.  Warren 

a 

DO  2  2 

c 

DO  9 

b 

91  SMW 

ISMD 

b 

Minot 

a 

D022 

b 

DO  9 

c 

308  SMW 

ISMD 

b 

Little  Rock 

c 

DO  2  2 

a 

321  SMW 

ISMD 

b 

Grand  Forks 

b 

DO  2  2 

a 

c 

341  SMW 

ISMD 

b 

Malmstrom 

a 

D022 

a 

DO  9 

b 

3  51  SMW 

ISMD 

b 

Whiteman 

b 

DO  2  2 

a 

DO  9 

a 

381  SMW 

ISMD 

b 

McConnell 

a 

a 

Seldom 


Seldom 


Unit 


Respondent  Comments 


44  SMW 
Ellsworth 


90  SMW 
F.E.  Warren 


91  SMW 
Minot 


308  SMW 
Little  Rock 


321  SMW 
Grand  Forks 


ISMD  Performed  when  new  61 6A  intro¬ 

duced;  also  performed  when  MCC 
maintenance  on  LCC  introduced 
DOTI  50-16  Vol  I  tasks  us  to  train 

each  task  on  annual  basis  - 
we  determine  at  this  step  how 
to  maintain  proficiency  and 
document  in  local  plans 
D022  CPOs  and  a  unit  Weapon  System 

Programmed  Text 
DO 9  JPRs 

ISMD  CPOs  are  result 

DOTI  LCC  maintenance  had  tasks  in 

Step  2  accomplished  prior  to 
introduction  of  training 
DO 2 2  CPOs 

D09  SAC  has  set  up  standards  in 

regulations  and  CPOs 

ISMD  LCC  maintenance  training 

DOTI 

D022  CPOs 

DO 9  Provided  to  us  in  SACR  55-56, 

Vol  I  in  the  form  of  JPRs 

ISMD  CPOs  and  JPRs;  at  unit  have  had 

Step  2  performed  on  transition 
training  program 
DOTI  CPOs  and  JPRs 

DO  2  2 

ISMD  CPOs;  at  unit,  Step  2  tasks 

performed  on  LCC  maintenance 
DOTI  CPOs,  Plan  of  Instruction, 

Local  training  regulations 
DO 2 2  CPOs;  SACR  50-21  and 

SACR  50-16,  Vol  I 
DO  9  JPRs 


Respondent  Comments 


341  SMW  ISMD 

Malmstrom 

DOT  I 

DO  2  2 
DO  9 

3  51  SMW  ISMD 

Whiteman 

li 

DOT  I 
DO  2  2 
DO  9 

381  SMW  ISMD 

McConnell 

DOT  I 
DO  2  2 


Produce  instruction  standards 
above  CPOs 

We  make  CPOs  into  specific 
standards  in  local  plans 
CPOs 

Incorporation  of  JPRs 

SAC  created  JPRLs  and  CPOs;  at 
unit:  look  at  availability  of 

classroom  facilities,  shelf  life 
considerations,  manpower 
resources  and  document  on  local 
plans 

CPOs  and  JPRs 
New  TO  and  tasks 
Master  Lesson  Plan 

Step  2  tasks  performed  on  MCCC 
transition  training 
CPOs,  DOT I  OIs,  special  and 
supplemental  lesson  plans 
CPOs 


7.  Does  your  unit  use  written  objectives  as  the  basis  for 
training? 


a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


Unit 


Respondent  Answer  Comments 


44  SMW 

ISMD 

a 

Ellsworth 

DOT  I 

a 

DO  2  2 

a 

DO  9 

a 

90  SMW 

ISMD 

a 

F.E.  Warren 

DOT  I 

a 

D022 

a 

DO  9 

a 

91  SMW 

ISMD 

a 

Minot 

DOT  I 

a 

D022 

a 

DO  9 

a 

308  SMW 

ISMD 

a 

Little  Rock 

DOTI 

a 

D022 

a 

321  SMW 

ISMD 

a 

Grand  Forks 

DOTI 

a 

D022 

a 

DO  9 

a 

341  SMW 

ISMD 

a 

Malms trom 

DOTI 

a 

DO  22 

a 

DO  9 

a 

3  51  SMW 

ISMD 

a 

Whiteman 

DOTI 

a 

DO  2  2 

a 

DO  9 

a 

381  SMW 

ISMD 

a 

McConnell 

DOTI 

a 

DO  22 

a 

Always 


Always 


Always 


Always 


8.  Written  tests  at  your  unit  are  based  upon  specific 
objectives . 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don ' t  Know 


Unit 

Respondent 

Answer 

Comments 

44  SMW 

ISMD 

a 

Always 

Ellsworth 

DOT  I 

a 

D022 

b 

DO  9 

b 

90  SMW 

ISMD 

a 

F.E.  Warren 

DOT  I 

a 

D022 

a 

DO  9 

a 

91  SMW 

ISMD 

a 

Minot 

DOTI 

a 

Always 

D022 

a 

DO  9 

a 

308  SMW 

ISMD 

a 

Always 

Little  Rock 

DOTI 

a 

D022 

a 

321  SMW 

ISMD 

a 

Grand  Forks 

DOTI 

a 

D022 

a 

DO  9 

a 

341  SMW 

ISMD 

a 

Malmstrom 

DOTI 

b 

Not  as  often  as 
should  be 

DO  2  2 

a 

DO  9 

a 

1  1 

£ 

•r: 

10.  Please  cite  examples 

of  criterion  tests  used  by  your  unit 

1 

<r 

which  require 

actual  demonstration  of  the  student's  performance. 

a 

Unit 

Respondent 

Commen  ts 

r.  - 

44  SMW 

ISMD 

MPT  scripts;  CPOs,  however. 

Ellsworth 

limit  much  creativity  -  though 

DOTI  can  go  beyond  CPOs 

DOT  I 

Remedial  training  lesson  plans; 

students  demonstrate  performance 

in  MPT 

", 

D022 

MPT  scripts.  Also  1  test  per 

quarter  must  comply  with  CPOs. 

2  other  tests  per  quarter 

DO  9 

conform  with  what  my  division 
feels  is  required  beyond  CPOs 
for  sufficient  knowledge 

MPT  scripts 

S1 

9  0  SMW 

ISMD 

MPT  scripts;  mock-up  scripts  for 

«■  _ 

F.E.  Warren 

individual  and  remedial 
training;  Many  examples  of  self- 
paced  programs 

DOT  I 

MPT  scripts;  LCC  maintenance 

demonstrations 

DO  2  2 

MPT  scripts;  tape-testing 

1  *  • 

DO  9 

MPT  scripts 

91  SMW 

ISMD 

MPT  scripts;  learning  center 

s.*  •- 

Minot 

(mock-up)  scripts  for  remedial 

H 

DOT  I 

training;  biopack  training 

MPT  scripts 

DO  2  2 

MPT  scripts;  learning  center 

scripts 

DO  9 

MPT  scripts 

•  *  • 

308  SMW 

ISMD 

Learning  center  scripts  for 

ia 

Little  Rock 

DOT  I 

individuals;  biopack  training; 
weather  corridor  plotting 

MPT  scripts;  platforms  and 
escape  hatch  lesson  plans  teach, 
then  have  student  demonstrate; 

rm 

t  t  . 

all  other  training  is  knowledge 
trai  ning 

» 

DO  2  2 

MPT  scripts 

> 

Sv- 
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Unit 

Respondent 

Comments 

321  SMW 

ISMD 

MPT  scripts;  learning  center 

Grand  Forks 

tests;  test  on  demonstrating 
proper  donning  of  Biopack  45 

DOT  I 

MPT  scripts;  individual  remedial 
training  covering  topic  areas 
(i.e.  security,  power,  etc) 

DO  2  2 

MPT  scripts,  standboard 
evaluations;  quarterly  CPO  test 

DO  9 

MPT  scripts 

341  SMW 

ISMD 

MPT  evaluation  scripts 

Malmstrom 

DOT  I 

MPT  scripts;  lesson  plans  which 
require  students  to  demonstrate 
( improvement  program  lesson 
plans ) 

DO  22 

MPT  scripts;  simulators  other 
than  MPT;  tape  testing 

DO  9 

MPT  scripts;  tests  on  mainten¬ 
ance  at  the  LCC 

3  51  SMW 

ISMD 

Unit  orientation  training  - 

Whiteman 

students  stand  by  equipment 
and  explain  what  they  would  do 

DOTI 

MPT  scripts 

DO  2  2 

MPT  scripts;  on-site  training  - 
perform  tasks  on  actual  equip¬ 
ment 

DO  9 

MPT  scripts;  computer-assisted 
instruction 

38  1  SMW 

ISMD 

MCCC  transition  training;  MPT 

McConnell 

scripts;  learning  center 
scripts;  training  on  guidance 
sys  terns 

DOTI 

MPT  scripts;  training  alerts 
where  students  demonstrate 
proficiency  on  actual  equipment; 
evaluations  (DOV) 

DO  2  2 

MPT  scripts;  quarterly  EWO  exam 
tape  tests 

152 


11.  Does  your  unit  make  an  effort  to  sequence  objectives  and 
instruction  in  a  manner  which  is  felt  to  best  promote  student 
learning? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


Unit 


Respondent 


Answer 


44  SMW 

ISMD 

a 

Ellsworth 

DOT  I 

a 

DO  2  2 

b 

DO  9 

b 

90  SMW 

ISMD 

b 

F.E.  Warren 

DOTI 

a 

D022 

a 

DO  9 

a 

91  SMW 

ISMD 

b 

Minot 

DOTI 

a 

DO  2  2 

b 

DO  9 

a 

308  SMW 

ISMD 

a 

Little  Rock 

DOTI 

a 

DO  2  2 

a 

321  SMW 

ISMD 

b 

Grand  Forks 

DOTI 

a 

DO  2  2 

b 

DO  9 

b 

341  SMW 

ISMD 

a 

Malms trom 

DOTI 

b 

a 

a 

Comments 


SACR  50-16  mandates 
each  JPRL  be 
covered  at  least 
once  a  year 


Regulations  limit 
sequencing  somewhat 


Limited  by  regula¬ 
tions  (must  cover 
each  JPRL  annually) 


Sometimes  sequence 
directed  by 
schedules,  etc 


Unit 

Respondent 

Answer 

Comments 

3  51  SMW 

ISMD 

a 

Whiteman 

DOT  I 

a 

D022 

a 

DO  9 

b 

38  1  SMW 

ISMD 

a 

McConnell 

DOT  I 

b 

Training  often  driven 
by  student  failures, 
so  we  frequently 
don' t  have  control 

over  sequencing  as 
we' d  like  to 

DO  22 


a 


At  your  unit : 


Student  self-pacing  methods  are  used  more  than 
lecture 

Student  self-pacing  and  lecture  are  used  equally 
Lecture  is  used  more  than  student  self-pacing 
Don’ t  Know 


Unit 


Respondent 


Answer 


Comments 


44  SMW 

ISMD 

b 

Self  study  on  complex 

Ellsworth 

DOTI 

b 

and  lecture  in 
classroom  every 
month;  self  study 
prepares  students  for 
class,  then  take  a 
pre-test  -  if  student 
scores  100%  he 
doesn' t  take  class 
3-tier  system  - 
self-study,  lecture 
MPT 

DO  2  2 

c 

DO  9 

c 

90  SMW 

ISMD 

c 

Can't  rely  on  self¬ 

F.E.  Warren 

pacing  to  great 
extent  because  of 

criticality  of  work 

DOTI 

c 

DO  2  2 

b 

DO  9 

b 

91  SMW 

ISMD 

b 

Remedial  training 

Minot 

DOTI 

b 

self-paced;  classroom 
training  is  lecture 

DO  2  2 

b 

DO  9 

b 

308  SMW 

ISMD 

c 

Little  Rock 

DOTI 

c 

• 

DO  2  2 

c 

321  SMW 

ISMD 

c 

Grand  Forks 

DOTI 

c 

Unit 

Respondent 

Answer 

Comments 

341  SMW 

ISMD 

b 

Self-paced  complex 

Malmstrom 

DOTI 

b 

exercises,  lecture 
grou  ps 

Formal  instruction  is 
lecture;  individual 
instruction  is  self- 
paced 

DO  2  2 

c 

DO  9 

c 

Regulations  require 
lecture 

3  51  SMW 

ISMD 

b 

Monthly  classroom 

Whiteman 

DOTI 

b 

lectures  limited  to 
groups  of  12  people 
-  seminar  and  discus¬ 
sion  encouraged 

DO  2  2 

b 

DO  9 

b 

38  1  SMW 

ISMD 

b 

McConnell 

DOTI 

c 

D022 

c 

SAC  doesn't  allow 
much  self-paced  EWO 
training;  regulations 
prescribe  classroom 
sessions  for  definite 

time  periods 

13.  Please  cite  examples  of  methods  in  which  student 
participation  is  encouraged  within  the  training  programs  of  your 


Respondent 


Comments 


44  SMW 
Ellsworth 


ISMD 


DOT  I 


DO  2  2 


90  SMW 
F.E.  Warren 


ISMD 


DOT  I 


DO  22 


Students  fill  out  critiques 
after  each  MPT  and  lecture; 
monthly  videotape  in  preclass 
provokes  questions;  prepare 
questions  prior  to  monthly 
lecture 

Seminar  approach;  MPT ,  mock-ups, 
remedial  one-on-one  training 
Message  copying  exercises; 
programmed  texts  on  launch 
reporting  and  communications 
monitoring 

Classroom  lectures  have  numerous 
questions  for  students 
Scenarios  presented  to  students 
during  lecture  -  asked  questions 
Questions  to  students  during 
lectures;  programmed  texts 
Informal  lectures  with  guided 
discussion 


91  SMW 
Minot 


ISMD 


DOT  I 
DO  2  2 


Learning  center  (mock-ups)  on 
voluntary  basis;  but  there  are 
time  and  manning  limits 
MPT  scripts 

Message  copying  exercise;  guided 
quest  ion -answer  session  during 
lectures 

Encourage  students  to  get  up  in 
front  of  class  and  answer 
specific  questions 


DO  9 


Unit  Respondent  Comments 


308  SMW  ISMD 

Little  Rock 

DOT  I 


D022 


321  SMW  ISMD 

Grand  Forks  DOTI 

DO  22 
DO  9 

341  SMW  ISMD 

Malmstrom 

DOTI 

DO  22 

DO  9 

3  51  SMW  ISMD 

Whiteman 

DOTI 
DO  2  2 


Participative  question  session 
during  monthly  training  (person 
from  audience  used) 

Transition  training  from  DMCCC 
to  MCCC ;  participative  question 
session  during  monthly  training; 
students  demonstrate  proficiency 
on  platforms  and  escape  hatch; 
MPT 

Student  inputs  used  as  feedback 
to  clarify  misconceptions 
( critiques ) 


Student  centered,  scenario- 
based,  guided  discussions  during 
monthly  classroom  training 
Self-improvement  programs  -  both 
on  voluntary  and  mandatory  basis 
Guided  question  -  answer  session 
during  lectures 

Scenarios  with  question  and 
answers  in  classroom;  EWO 
message  copying  exercises 
Improvement  program  gets  people 
to  take  additional  training; 
scenarios  with  question  and 
answer  sessions 
Volunteer  program  for  extra 
training;  direct  questions 
during  lectures 
Discussion  and  demonstration 
during  lecture 

Complex  self-study  packages, 
individual  improvement  packages, 
monthly  recurring  classrooms  use 
seminar  and  discussion 
Seminar  format  in  monthly  class¬ 
room  training 

Open-ended  questions;  seminar 
classrooms  in  DOT  &  D022; 
many  programmed  texts  for  self- 
study 

Monthly  self-study  guide  (self- 


009 


381  SMW 
McConnell 


ISMD 


DOT  I 


MPT  scripts;  learning  center 
scenarios 

Transition  training  -  has  cross 
talk  between  students  and 
between  them  and  instructor. 
Feedback  also  a  big  part. 

FMIS  reporting  exercises;  USKAC 
72  message  exercises;  tape 
testing 


DO  2  2 


14.  Does  your  unit  evaluate  the  advantages  and  disadvantages  of 
various  media  when  developing  instruction? 


a. 

Often 

b. 

Sometimes 

c . 

Never 

d. 

Don' t  Know 

Respondent 


Answer 


Comments 


44  SMW 

ISMD 

a 

A  monthly  meeting  of 

Ellsworth 

DOT  I 

a 

senior  persons  within 
training  and  eval 
branches  discusses 
new  items  for  train¬ 
ing  and  how  to  do  it 

DO  22 

b 

DO  9 

b 

Don't  have  time 

always  -  just  go  with 
what  we  did  before 

90  SMW 

ISMD 

a 

F.E.  Warren 

DOT  I 

b 

Once  media  set, 
generally  same  for 
future  similar 
instruction 

DO  22 

b 

Sometimes  a  matter 
of  convenience,  as 
opposed  to  actually 
evaluating 

DO  9 

b 

__ 

91  SMW 

ISMD 

a 

Remedial  training 

Minot 

DOT  I 

b 

examined  always  to 
see  whether  it  is 
better  to  train  in 
learning  center  or 

MPT 

DO  2  2 

b 

DO  9 

b 

As  time  permits 

160 
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Respondent  Answer 


Comments 


308  SMW 
Little  Rock 


321  SMW 
Gr  nd  Fc ~ks 


341  SMW 
Malmstrom 


351  SMW 
Whiteman 


38  1  SMW 
McConnell 


ISMD  b 

DOT  I  a 

DO  2  2  a 

ISMD  b 

DOT  I  b 

DO  2  2  b 

DO  9  b 

ISMD  b 

DOTI  b 

D022  b 

DO  9  a 

ISMD  b 


DOTI  b 

DO  2  2  a 


DO  9  b 

ISMD  a 

DOTI  a 

DO  2  2  b 


We  often  try,  but 
resources  limit 
application  of  this 


Must  constantly  pump 
out  new  stuff  - 
deadlines  keep  you 
from  looking  at  this 
carefully 

Not  much  time  to 
worry  about  this 


Seldom.  Primarily 
lecture  or  discussion 
with  some  viewgraphs. 
Base  has  limited 
audio-visual 
capability 

Often  looked  at,  but 
not  considered  as 
much  as  should  be  - 
instructors  change 
too  much 


15.  Does  your  unit  perform  a  tryout  on  a  small  group  of 
students  in  order  to  validate  instruction? 


a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


Unit 

Respondent 

Answer 

Comments 

4  4  SMW 

ISMD 

b 

Ellsworth 

DOT  I 

c 

D022 

c 

DO  9 

c 

90  SMW 

ISMD 

b 

Recently  started 

F.E.  Warren 

DOT  I 

b 

tryout  of  DOV  script 
on  ordinary  crew 
(crew  not  held 
accountable ) 

DO  2  2 

c 

Time  doesn't  allow 

DO  9 

b 

91  SMW 

ISMD 

b 

LCC  maintenance 

Minot 

DOTI 

c 

training  had  tryout 

DO  2  2 

c 

DO  9 

c 

308  SMW 

ISMD 

b 

Little  Rock 

DOTI 

c 

DO  2  2 

c 

321  SMW 

ISMD 

c 

Not  practical 

Grand  Forks 

DOTI 

b 

DO  2  2 

c 

DO  9 

c 

341  SMW 

ISMD 

b 

Malms trom 

DOTI 

c 

Respondent 


Answer 


Comments 


3  51  SMW 

ISMD 

b 

Very  limited  tryouts 

Whiteman 

DOT  I 

c 

First  group  of 
students  to  take 
actual  instruction 
help  validate,  but 
students  are  held 
accountable 

D022 

b 

Seldom 

DO  9 

b 

38  1  SMW 

ISMD 

b 

McConnell 

DOT  I 

b 

16.  Please  cite  any  other  method  you  would  consider  as  a 
means  of  validating  instruction. 


Unit 


Respondent  Comments 


44  SMW 

ISMD 

Each  training  agency  has  all 

Ellsworth 

instructors  take  each  test; 
coordination  process  is  valid¬ 
ation;  Chief,  ISMD  performs 
each  MPT  prior  to  its  release 

DOT  I 

Proof  of  scripts  by  other 
instructors 

DO  2  2 

Tryout  of  tests,  instruction 
and  scripts  by  other  instructors 
and  DO  5 

DO  9 

Coordination  process;  tryout 
of  instruction  by  other 
instructors 

90  SMW 

ISMD 

Analysis  of  test  results 

F.E.  Warren 

DOT  I 

Tryouts  by  other  instructors; 
coordination  process 

DO  2  2 

Tryouts  by  own  instructors; 
coordination  process 

DO  9 

Coordination  process 

91  SMW 

ISMD 

Coordination  process;  DOTI, 

Minot 

D022  give  MPT  scripts  and  tests 
to  own  crews 

DOTI 

Student  critique  sheets; 
instructor  tryouts  of 
instruction  and  scripts 

DO  2  2 

Coordination  process;  tryout  of 
instruction  by  other  instructors 

DO  9 

Questions  that  are  continually 
high  miss  questions  mandate 
redoing  lesson  plan 

308  SMW 

ISMD 

Validated  by  trial  and  error  and 

Little  Rock 

by  tests  after  each  training 
session;  instructors  and 
evaluators  tryout  own  scripts 

DOTI 

A  test  after  each  recurring 
training  session;  Instructors 
and  evaluators  tryout  MPT 
scripts;  coordination  process 
validates  accuracy,  but  not 
effectiveness 

Unit 


Respondent  Comments 


321  SMW  ISMD 

Grand  Forks  DOTI 

0022 
DO  9 


341  SMW  ISMD 

Malmstrom 

DOTI 


DO  2  2 
DO  9 


3  51  SMW  ISMD 

Whiteman 

DOTI 
DO  2  2 


DO  9 

38  1  SMW  ISMD 

McConnell  DOTI 


Present  training  to  other 
instructors 

Other  instructors  tryout 
instruction 

Coordination  process;  tryouts 
by  other  instructors  on  scripts 
and  tests 

Tryouts  by  other  instructors  and 
evaluators;  coordination 
process;  continuous  feedback 
process  with  students 
Tryouts  by  other  instructors; 
coordination  process 
Testing  and  code  handling 
procedures  in  field  are  fed 
back 

Coordination  process  also 
validation  somewhat 
Coordination  process;  tryout  of 
instruction  by  other  instructors 
Scripts  tried  out  by  instructors 
up  to  5  or  6  times.  Monthly 
lesson  plans  presented  in 
instructor  training.  Coordi¬ 
nation  is  validation 
Analysis  test  results 


Training  validated  by  actual 
student  performance 
Classroom,  MPT  observations; 
feedback  loop 


DO  2  2 


17.  How  does  your  unit  conduct  internal  evaluation  on  its 
training  programs? 


Unit 


44  SMW 
Ellsworth 


90  SMW 
F.E.  Warren 


91  SMW 
Minot 


308  SMW 
Little  Rock 


Respondent  Comments 


ISMD 


DOTI 
DO  2  2 

DO  9 


DOV  checks  on  crews;  STRP  meet¬ 
ings  every  quarter  review  all 
deficiencies  and  where  and  how 
to  retrain 

Quarterly  review  of  lesson 
plans;  tests;  DOV  evaluations 
Monthly  tests;  feedback  from 
self-study  exercises;  DOV 
evaluations 
Tests 


ISMD 

DOTI 

DO  22 

DO  9 

ISMD 

DOTI 

DO  2  2 
DO  9 

ISMD 

DOTI 


Analysis  of  test  results 
Look  at  results  of  testing  and 
DOV  evaluation 

Continuous  review  of  programs  by 
instructors;  review  of  error 
analysis;  DOV  checks; 
observations  of  crews 
Coordination  process 

Triggered  by  deficiencies  from 
evaluations  and  tests 
DOV  checks;  periodic  review  of 
all  training 

Tests  and  DOV  evaluations 
Analysis  of  test  results  - 
continuous  high  miss  items 
mandate  redoing  of  lesson  plans 

Evaluations;  monthly  error 
analysis;  quarterly  STRP; 
analysis  of  monthly  tests 
By  analyzing  monthly  tests; 
through  recurring  evaluations 
by  DOV 

Tests  and  DOV  evaluations 


DO  2  2 


Unit 


Respondent 


Comments 


321  SMW 

ISMD 

Tests;  standardization 

Grand  Forks 

evaluations  -  student  must 
demonstrate  performance 

DOT  I 

Error  analysis  on  high  missed 
tasks  and  questions  from 
evaluations  and  tests 

DO  2  2 

Tests  and  DOV  evaluations 

DO  9 

Tests 

341  SMW 

ISMD 

Coordination;  instructor 

Malms trom 

tryouts;  also  observations  of 
instructors;  student  feedback 

DOTI 

Train  concepts  a  month  or  two 
before  putting  in  MPT  script; 
DOV  evaluations 

DO  2  2 

Periodic  review  of  trends; 
student  feedback;  DOV  checks 

DO  9 

Testing  and  actual  performance 

3  51  SMW 

ISMD 

Each  student  tested  after 

Whiteman 

classroom.  Long  term  retention 
tested  on  MPT. 

DOTI 

Periodic  review  boards  look  for 
trends  in  faulty  performance  or 
training 

DO  2  2 

Tests  on  lesson  plans;  DOV 
evaluation;  no  notice  codes  and 
EWO  tests 

DO  9 

Testing 

38  1  SMW 

ISMD 

Coordination  process 

McConnell 

DOTI 

Feedback  on  student  performance 
(tests,  MPT);  feedback  from 
other  unit  agencies  (squadrons, 
DOV,  DO 5,  D022) 

DO  2  2 

Feedback  loop  (tests,  MPT)  lets 
us  know  how  students  are  doing 
so  we  can  adjust  and  reaccomp¬ 
lish  when  necessary 

18.  Which  of  the  following  would  you  consider  external 
evaluation?  (More  than  one  response  may  be  appropriate) 

a.  Standardization/Training  Conferences 

b.  CPO  conferences 

c.  EWO  conferences 

d.  Higher  Headquarters  evaluations 

e.  Other: 


Unit 

Respondent 

Answer 

Comments 

44  SMW 

ISMD 

d  f  b  f  c  f  cl 

Ellsworth 

DOT  I 

a,b ,d,e 

Staff  assistance 
visits;  cross-talks 
between  units 

DO  2  2 

d,  e 

Staff  assistance 
visits 

DO  9 

d,e 

Codes  conference 

90  SMW 

ISMD 

Q) 

Locally  developed 

F.E.  Warren 

DOTI 

a,b,c,d,e 

exercises;  unit 
feedback  program 

Own  local  evaluations 
i.e.  -  local  stand- 
board  and  ORIs 

D022 

a,b,c,d,e 

Self -inspect ion 
program;  staff 
assistance  visits 

DO  9 

a,d,e 

Codes  conference 

91  SMW 

ISMD 

d,e 

Local  exercises 

Minot 

DOTI 

a,  bfd 

DO  2  2 

a,b,c,d,e 

Staff  assistance 
visits  and  local 
exercises 

DO  9 

d 

308  SMW 

ISMD 

d,e 

AF  Assistance 

Little  Rock 

DOTI 

a,b ,d ,  e 

visit;  dignitary 
visits;  staff  officer 
comments;  Olympic 
plays 

Reviewing  task 
analysis  from  3901st 
quarterly  pamphlet; 
inputs  to  SMES 
through  clarification 

messages 

DO  2  2 

a ,  b 

168 


Unit 


Respondent  Answer  Comments 


321  SMW  ISMD 

Grand  Forks  DOTI 

0022 
DO  9 


341  SMW  ISMD 

Malms trom 


DOTI 
DO  2  2 

DO  9 

351  SMW  ISMD 

Whiteman 


DOTI 
DO  2  2 

DO  9 

381  SMW  ISMD 

McConnell  DOTI 

DO  2  2 


d 

a 

a,b,c,d 

d,e  Codes  conferences; 

staff  assistance 
visits 


df b 9 c f d  /  0 

DOV  an  external 
source  -  because  they 
aren't  a  training 
agency;  Standardiza¬ 
tion-Review  Panel 

s  t  b  f  d  f  © 

DOV  in  some  respects 
external  evaluation 

b, c,d, 0 

Required  to  send  much 
instruction  material 
to  HHQ 

d ,e 

Codes  conferences; 
no  possible  code 
compromises 

dr  e 

Supervisors  (i.e. 
commanders , 
operations  officers) 
offer  unsolicited 
comments;  4315th 

CCTS  sends  out 
questionnaires 

3.  f  b  f  c  t  d  f  0 

Staff  assistance 
visits 

3  f  b  f  c  f  d 

Staff  assistance 
visits;  local 
exercises , 

supervisors  comments 

a,b,c,d 

Codes  conferences 

d 

a, b,c,d,e  Local  exercises; 

staff  assistance 
visits 

b, c 


L  69 


19.  Does  your  unit  perform  goal  analysis  when  developing 
training  programs? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


Unit 

Respondent 

Answers 

Comments 

44  SMW 

ISMD 

b 

616A  modification 

Ellsworth 

goals;  but  CPOs 
generally  preempt 
this  -  primary  goal 
to  train  IAW  CPOs  - 

only  do  for  new 
equipment 

DOT  I 

b 

Goes  hand  in  hand 
with  determining 
method  of  training 

DO  2  2 

b 

— 

DO  9 

b 

Seldom 

90  SMW 

ISMD 

b 

Everyone  wants  to 

F.E.  Warren 

DOTI 

b 

spark  up  interest  by 
using  something 
different 

Officers  are  impli¬ 
citly  expected  to 
achieve  unwritten 
goals 

Don't  specifically 

DO  2  2 

b 

say  these  are  our 
goals 

DO  9 

d 

91  SMW 

ISMD 

d 

Minot 

DOTI 

b 

DO  2  2 

c 

DO  9 

b 

Seldom 

308  SMW 

ISMD 

b 

Little  Rock 

DOTI 

b 

Rarely  -  no  specific 
examples  come  to  mind 

Unit 


Respondent  Answer 


Comments 


321  SMW 

ISMD 

b 

Grand  Forks 

DOT  I 

b 

DO  2  2 

c 

DO  9 

b 

341  SMW 

ISMD 

b 

Malms trom 

DOT  I 

a 

DO  2  2 

c 

* 

DO  9 

a 

351  SMW 

ISMD 

b 

Whiteman 

DOT  I 

b 

DO  22 

b 

DO  9 

d 

Pretty  much  common 
sense  approach 

Want  to  change 
attitudes  from  rote 
memorization  to 
understanding 

No  official  goal 
analysis,  but  each 
division  has  own 
goals 
Seldom 


38  1  SMW 
McConnell 


ISMD 
DOT  I 
DO  22 


b 

b 

b 


Seldom 


20.  Does  your  unit  present  instruction  in  the  manner  recommended 
by  Mager? 

a.  Often 

b.  Sometimes 

c.  Never 

d .  Don' t  Know 


Unit 


Respondent  Answer  Comments 


44  SMW 

ISMD 

b 

Crew  member  enrich¬ 

Ellsworth 

DOTI 

b 

ment  program  - 
student  gets  pretest 
and  post-test,  then 
course  structured 
on  his  needs;  same 
structure  for 

Quality  Improvement 
Program 

DO  2  2 

b 

DO  9 

b 

90  SMW 

ISMD 

b 

Impractical  for  tests 

F.E.  Warren 

DOTI 

b 

to  cover  only 
objectives;  instead 
take  questions  from 
outside  objectives  - 
must  be  ready  for 
anything 

Mainly  used  for 
students  having 
problems 

DO  2  2 

a 

Student  sees 

objectives  right  at 
beginning  and  gets 
plenty  of  practice 
at  each  training 
session 

DO  9 

a 

91  SMW 

ISMD 

a 

For  complex  self- 

Minot 

study  training 

DOTI 
DO  2  2 
DO  9 


c 

b 

a 


This  is  all  in  MCCMs' 


Unit 


Respondent  Answer 


Comments 


308  SMW 

ISMD 

c 

Little  Rock 

DOT  I 

b 

Transition  training 
to  MCCC  is  an  example 

DO  2  2 

a 

321  SMW 

ISMD 

b 

Grand  Forks 

DOT  I 

c 

DO  2  2 

c 

DO  9 

b 

341  SMW 

ISMD 

a 

Self-study  -  class¬ 

Malmstrom 

DOT  I 

b 

room  -  MPT  process  is 
pretty  much  same  as 

Mager' s  recommend¬ 
ation 

DO  2  2 

b 

DO  9 

c 

3  51  SMW 

ISMD 

b 

Whiteman 

DOT  I 

b 

DO  2  2 

a 

Can' t  always  self¬ 
pace,  but  always 
other  things  to 
tailor  to  student 

DO  9 

b 

38  1  SMW 

ISMD 

b 

McConnell 

DOT  I 

b 

DO  2  2 

a 

Complex  training 
package 

21.  The  CPOs  provide  a  beneficial  basis  for  training  and 
evaluation . 


a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


Unit  Respondent  Answer  Comments 


44  SMW 

ISMD 

a 

Ellsworth 

DOT  I 

b 

DO  2  2 

c 

DO  9 

b 

90  SMW 

b 

F.E.  Warren 

b 

DO  2  2 

b 

DO  9 

b 

91  SMW 

ISMD 

a 

Minot 

DOT  I 

b 

DO  2  2 

b 

DO  9 

b 

308  SMW 

ISMD 

a 

Little  Rock 

a 

D022 

b 

321  SMW 
Grand  Forks 


ISMD  b 

DOT  I  b 

DO  2  2  a 

DO  9  b 


341  SMW 

a 

Malmstrom 

b 

DO  2  2 

a 

DO  9 

b 

3  51  SMW 
Whiteman 


ISMD  b 

DOT  I  b 

DO  2  2  a 

DO  9  b 


P 


P 


Unit 


Respondent  Answer 


Comments 


381  SMW  ISMD  d 

McConnell 


DOT  I  a 

DO  2  2  b 


Beneficial  for 
evaluation  only;  CPOs 
also  not  student 
oriented  -  they  pin 
down  performance  only 
in  general  terms  and 
are  not  sufficient 
for  design  or 
operation  of  training 


22.  Improved  communication  between  DOV  and  the  training 
divisions  is  one  benefit  of  the  CPOs. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


Unit 

Respondent 

Answer 

44  SMW 

ISMD 

a 

Ellsworth 

DOT  I 

b 

D022 

c 

DO  9 

b 

90  SMW 

ISMD 

b 

F.E.  Warren 

DOT  I 

b 

DO  2  2 

b 

DO  9 

a 

91  SMW 

ISMD 

b 

Minot 

DOT  I 

b 

DO  22 

b 

DO  9 

c 

308  SMW 

ISMD 

b 

Little  Rock 

DOT  I 

b 

D022 

a 

321  SMW 

ISMD 

* 

c 

Grand  Forks 

DOT  I 

a 

DO  22 

e 

DO  9 

b 

341  SMW 

ISMD 

a 

Malmstrom 

DOT  I 

a 

DO  2  2 

b 

DO  9 

c 

351  SMW 

ISMD 

c 

Whiteman 

DOT  I 

b 

DO  2  2 

b 

DO  9 

c 

38  1  SMW 

ISMD 

c 

McConnell 

DOT  I 

d 

DO  2  2 

a 

23.  CPOs  have  accurate  and  appropriate  time  standards 


a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


Unit  Respondent  Answer  Comments 


44  SMW 

ISMD 

c 

There  is  no  guidance 

Ellsworth 

as  to  why  certain 
standards  are  like 

they  are?  there 
should  be  reassess¬ 
ment  of  level  4's  not 
constrained  by  weapon 
system 

DOT  I 

e 

DO  2  2 

d 

* 

DO  9 

b 

9  0  SMW 

ISMD 

b 

F.E.  Warren 

DOT  I 

b 

DO  2  2 

c 

DO  9 

b 

91  SMW 

ISMD 

b 

Minot 

DOT  I 

DO  2  2 

c 

DO  9 

d 

308  SMW 

ISMD 

c 

Little  Rock 

DOT  I 

b 

Some  too  lenient, 
some  too  strict 

however 

DO  2  2 

b 

321  SMW 

ISMD 

c 

Grand  Forks 

DOT  I 

c 

DO  2  2 

e 

DO  9 

b 

341  SMW 

ISMD 

d 

Malms trom 

DOT  I 

c 

Many  cases  good, 
many  cases  not 

Unit 

Respondent 

Answer 

Comments 

3  51  SMW 

ISMD 

b 

Whiteman 

DOT  I 

DO  22 

d 

Not  all  accurate 

DO  9 

b 

38  1  SMW 

ISMD 

a 

McConnell 

DOT  I 

b 

DO  22 

c 

Some  accurate,  some 

not  ' 


24.  CPOs  inhibit  trainer/evaluator  judgement  on  the  MPT 


a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disagree 

e.  Strongly  Disagree 


Unit 


Respondent  Answer  Comments 


44  SMW 

ISMD 

d 

Ellsworth 

DOT  I 

a 

DO  2  2 

b 

DO  9 

b 

9  0  SMW 

ISMD 

e 

F.E.  Warren 

DOT  I 

b 

D022 

d 

DO  9 

a 

91  SMW 

ISMD 

a 

Minot 

DOT  I 

b 

DO  2  2 

b 

DO  9 

b 

308  SMW 

ISMD 

d 

Little  Rock 

DOT  I 

a 

DO  2  2 

b 

321  SMW 

ISMD 

b 

Grand  Forks 

DOTI 

a 

DO  2  2 

a 

DO  9 

d 

341  SMW 

ISMD 

b 

Malmstrom 

DOTI 

a 

DO  2  2 

d 

DO  9 

c 

3  51  SMW 

ISMD 

b 

Whiteman 

DOTI 

d 

DO  2  2 

b 

DO  9 

a 

381  SMW 

ISMD 

e 

McConnell 

DOTI 

d 

D022 

e 

Lack  of  proficiency  easy 
to  recognize/it  constrains 
subjectivity,  not  inhibits 


Evaluator  now  scorekeeper 
rather  than  an  evaluator 


Agree,  but  that  is 
positive  -  there  should  be 
limited  judgement 


But  considers  this  a 
benefit,  not  a  problem 


25.  CPOs  convey  the  wrong  signal  by  emphasizing  speed  rather 
than  correct  action. 

a.  Strongly  Agree 

b.  Agree 

c.  Uncertain 

d.  Disag  re'e 

e.  Strongly  Disagree 


Unit 


Respondent  Answer  Comments 


44  SMW  ISMD 

Ellsworth  DOTI 

D022 
DO  9 


d 

b 

d 

d 


90  SMW  ISMD 

F.E.  Warren  DOTI 

DO  2  2 
DO  9 


a 

d 

d 

a 


91  SMW 
Minot 


ISMD  a 


DOTI  b 

D022  d 

DO  9  d 


Often  hear  crews 
comment  at  outset  of 
a  problem  on  the  MPT 
on  how  long  they 
have,  not  vrtiat  they 
h  ave 


308  SMW 
Little  Pock 


ISMD 

DOTI 

D022 


d 

d 

d 


321  SMW 
Grand  Forks 


ISMD 
DOTI 
D022 
DO  9 


c 

d 

e 

d 


341  SMW  ISMD 

Malms trom  DOTI 

D022 
DO  9 


b 

a 

d 

b 


180 


Unit 


Respondent 


Answer 


Conanen  ts 


3  51  SMW 

ISMD 

b 

Crews  skip  steps 

Whiteman 

DOTI 

d 

because  of  improper 
emphasis  on  speed 

D022 

d 

D09 

b 

381  SMW 

ISMD 

e 

McConnell 

DOTI 

d 

D022 

d 

V  V 


y  - ' 


m 

rr^-r 


H 


„  *  _  wj'ir.  ^  V.  fcf'J  *  J  MT .  'ir  .  ».  rJ"m\rMm 


26.  Are  there  any  other  benefits  or  problems  with  the  CPOs  that 
you  would  care  to  discuss? 


Unit 


Respondent  Comments 


44  SMW  ISMD 

Ellsworth 

DOTI 

D022 

DO  9 


Problem:  CPOs  necessary/but 

structure  should  be  changed  for 
more  flexibility  and  realism 
Problem:  CPOs  should  only  be  a 
guide  for  training?  but  for 
evaluation  they  are  proper?  can 
be  improved 

Problem:  Doesn't  list  all 

required  actions  to  be 
evaluated 


90  SMW  ISMD 

F.E.  Warren  DOTI  Problem:  Students  dwell  more  on 

CPOs  than  on  task  itself.  They 
learn  only  as  much  as  they  have 
to 

D022  Problem:  MCCMs  often  more 

concerned  with  CPO  than  task 
itself?  CPOs  not  tested  or 
validated  prior  to  publishing 

DO  9 


91  SMW 
Minot 


ISMD  Problem:  No  training  really 

goes  beyond  CPOs  any  more 
DOTI  Benefits:  Provides  basic 

expectations  to  MCCMs  and  good 
outline  for  lesson  plans? 
Problem:  Inhibit  total  concept 

understanding?  create  artificial 
environment  and  standards 

D022 
DO  9 


308  SMW  ISMD 

Little  Rock 


DOTI 


DO  2  2 


Benefit:  Standardize  training 
and  evaluation 

Problem:  Task  coding  forces 

some  to  get  error  even  when  not 
in  position 

Problem:  Existence  of  CPOs 

limits  training  and  knowledge  - 
people  rarely  go  beyond  what 
CPOs  require 

Benefit:  Standardization 


Vjil*. 


vL  .■l»  _a . 


l 
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Unit 


Respondent  Comments 


321  SMW 
Grand  Forks 


341  SMW 
Malms trom 


3  51  SMW 
Whiteman 


ISMD  Benefit:  Provides  black  and 

white  guidance 

Problem:  Can  inhibit  training  - 
it  is  necessary  for  SAC  to 
mandate  training  above  standards 
of  CPOs 

DOTI 

D022  Benefit:  Sound  foundation  of 

performance  requirements 
Problem:  Promotes  mediocrity  - 
crews  know  just  enough  to  get  by 

DO  9 


ISMD 

DOTI 

DO  2  2 

DO  9 

ISMD 


DOTI 

D022 

DO  9 


Problem:  Puts  too  much  emphasis 
on  things  crew  members  do 
seldom;  don't  update  CPOs  fast 
enough  when  a  change  needed 
Problem:  Try  to  be  all 
inclusive,  but  they  are  not  - 
leaves  problem  of  interpretation 
Problem:  Same  penalty  for  not 

doing  a  task  at  all,  as  for 
missing  CPO  timing 
Problem:  People  interpret  CPOs 

in  different  ways 

Problem:  Stress  performance  at 

expense  of  knowledge  and  skills 
-  emphasis  on  time,  but  actually 
should  emphasize  prompt  skillful 
action;  CPOs  don't  allow  imagi¬ 
nation  of  instructors  -  because 
of  criticality  of  tasks  and  time 
constraints 

Benefit:  CPOs  a  self-study 
guide  for  crew  member  himself 


18  3 


Unit 


Respondent  Comments 


381  SMW  ISMD  Benefit:  Standardizes  evalua- 

McConnell  tion  and  training 

Problem:  Still  too  much  room 
for  personal  interpretation 
DOTI  Benefit:  Objective  standards, 

not  based  on  judgement  of 
evaluator;  Leaves  little  doubt 
about  what's  expected  of  student 
Problem:  Sometimes  don't  cover 

enough  contingencies;  Not 
updated  frequently  enough 
D022  Benefit:  Standardize  training  & 

evaluat  ion 

Problem:  CPOs  not  designed  to 
evolve  with  changes  in  SAC 
philosophy  -  lag  behind  real 
world  -  create  more  questions 
than  answers 


27.  Which  of  the  following  statements  reflect  your  opiniion  of 
the  benefits  of  using  ISD  at  the  operational  missile  unit. 

a.  Student  performance  has  increased. 

b.  Student  learning  rates  have  improved. 

c.  Student  motivation  has  improved. 

d.  Total  training  time  is  reduced. 

e.  Costs  are  reduced. 

f.  ISO  provides  definite,  yet  flexible  guidelines  for 
developing  and  conducting  instruction. 

g.  I  don't  see  any  benefits  in  using  ISD  at  my  unit. 

h .  Other  _  _ _ 


Unit 

Respondent 

Answer  Comments 

44  SMW 

ISMD 

a,b,c,f,h  h:  Because  it  has 

Ellsworth 

mandated  CPOs,  it 
allows  crew  member  to 
focus  study  time 

DOTI 

b ,  f 

D022 

9 

DO  9 

f 

90  SMW 

ISMD 

d,  f 

F.E.  Warren 

DOTI 

a,b,c,d,e,f 

D022 

c,d,e,f 

DO  9 

b,c ,d,e,f 

91  SMW 

ISMD 

a,c ,  f 

Minot 

DOTI 

f 

DO  2  2 

a,b,d,f 

DO  9 

a,b,c,d,e,f 

308  SMW 

ISMD 

a,b,d,e,f 

Little  Pock 

DOTI 

g 

D022 

f 

321  SMW 

ISMD 

a,b,f,h  h:  Makes  us  look  at 

Grand  Forks 

entire  training 
program  to  see  if  it 
is  effective;  also 
increases  coordination 
between  DOTI,  D09, 

DOV,  D022 

DOTI 

f 

D022 

f 

DO  9 

3  f  b  f  d  /  f 

18  5 


Unit 

Res  pondent 

Answer 

Commen  ts 

341  SMW 

ISMD 

b,C/f 

Malms trom 

'  DOTI 

a,b,e,f 

DO  2  2 

e,f 

DO  9 

g 

351  SMW 

ISMD 

f 

Whiteman 

DOTI 

f 

D022 

b,d,e,f rh 

h:  A  common  focus  for 

DO  9 

b/  f 

training  and 
evaluation 

381  SMW 

ISMD 

a,b,d,f 

McConnell 

DOTI 

a,d,f  ,h 

h:  Enhances  instructor 
expertise;  involves 
the  student  more 

D022 

d,  h 

h;  Closes  the  loop  on 

training  to  provide 
feedback  to  improve 
areas  which  show 
weakness.  We  train 
what  is  needed  based 
on  statistical 
evidence 


28.  Which  of  the  following  reflect  your  opinion  of  the  proble 
caused  by  using  ISD  at  the  operational  missile  unit? 


a.  It  is  extremely  time-consuming  to  develop  training 
using  ISD. 

b.  Developing  concise,  understandable  objectives  is  a 
difficult  task. 

c.  Lack  of  cooperation  or  support  between  agencies 
hinders  the  process. 

d.  Constant  turnover  of  military  personnel  results  in 
instructors  who  lack  training  and  experience  in 
performing  ISD  tasks. 

e.  Higher  Headquarters  regulations  and  directives 
often  interfere  with  the  process. 

f.  The  seemingly  endless  series  of  changes  to  the 
weapon  system,  regulations,  and  technical  data 
hinders  the  ISD  process.  Much  time  and  money  is 
consumed  re-doing  tasks. 

g.  ISD-developed  training  programs  lack  sufficient 
theoretical  information  to  enable  long-term 
retention  (in  other  words,  students  can  absorb 
material  quickly,  pass  a  test,  then  forget  it 
qu  ickly)  . 

h.  I  don't  see  any  problems  using  ISD  at  my  unit. 

i  .  Other  _ _ _ _ _ 


Unit  Respondent  Answer  Comments 


44  SMW 
Ellsworth 


ISMD 


d,  i 


DOT  I 


c ,  d ,  f  ,  1 


For  d :  DMCCCs  can 
only  stay  instructors 
a  short  time;  as  soon 
as  someone  becomes 
useful  it  is  time  to 
leave. 

For  i:  Constraints 
on  Steps  1  and  2 
hinders  flexibility 
with  rest  of  process 
For  i;  People,  don't 
care  unless  it  is  in 
CPOs  and  evaluatable 


e 

d 


Unit 


Res ponden t 


Answer 


Commen  ts 


90  SMW 
F.E.  Warren 


91  SMW 
Minot 


308  SMW 
Little  Pock 


ISMD 
DOT  I 
DO  2  2 


DO  9 

ISMD 
DOT  I 


D022 
DO  9 


ISMD 
DOT  I 


DO  22 

ISMD 
DOT  I 


D022 
DO  9 


b  /C  ,d  f  e  t  f  ,  g 
c  ,d,f  ,g 

b,d,i  For  i:  Some  people 

more  concerned  with 
how  you  are  doing  it 
as  opposed  to  vdiat 
you  are  doing 

c  ,d , e,g 


a, c  r d  ,  e  t  f 

c,g,i  For  i:  Students 

perform  only  to  level 
of  CPO  stand ard 

a  t b r d  i  e  ,g 

b, d,e,f,i  No  real  focus  on 

recurring  training  - 
it  seems  like  once 
students  learn  a 
task,  training  is 
done 


a,b  ,c  ,d ,  f  ,g 

a, b,c  ,d  ,e 

f,g  For  e:  One  regulation 

generally  says  to  use 
ISD ,  but  other  regu¬ 
lations  restrict  it 
by  requiring  a 
particular  #  of  hours 
for  training 

d 

b, d 

a,c,d,e,i  For  i:  Too  many 

regulations  and  too 
much  emphasis  on 
numbers  of  hours  of 
training  rather  than 
quality;  inhibits 
effectiveness  of  ISD 

c, d  ,efg 

a  ,  b ,  c  ,  a  ,  e ,  g 
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4 


Unit 

Fes  pondent 

Answer  Comments 

341  SMW 

ISMD 

b  ,d 

Malms  trom 

DOT  I 

b,c ,d,e 

f,g,i  For  i:  Because  we 

are  not  a  training 
unit,  it  is  hard  to 
apply 

D022 

b,c  ,d,e,g 

DO  9 

a , b ,c  ,d , f  ,g 

3  51  SMW 

ISMD 

b,c,d,f,i  For  i:  Job  centered. 

Whiteman 

student  centered 
training  only 
provided  lip  service 
at  units 

For  f :  When  you  keep 
changing  what  someone 
learned  first  time, 
it  hinders  students 
(what  is  learned 
first  is  learned 
bes  t) 

DOT  I 

a,b,c,d,f  ,g 

D022 

a,b,c,d,f  ,g 

DO  9 

a  ,b  ,d ,  f  ,g 

381  SMW 

ISMD 

a,c  ,d 

McConnell 

DOT  I 

a,c  ,d,e 

f,g,i  For  i:  Results  gained 

may  not  be  worth 
effort  spent 

D022 

c,e,f,g  For  c:  Often  receive 

CPOs  later  than  vrtien 
you  need  them 
For  f:  Usually  have  1 
or  2  days  notice  for 


29.  This  concludes  the  structured  part  of  my  survey;  however,  I 
would  appreciate  any  other  comments  regarding  this  survey,  ISD  in 
general,  or  any  specific  aspects  of  ISD. 


Unit  Respondent  Comments 


44  SMW  ISMD 

Ellsworth 


DOT  I 
DO  2  2 


DO  9 

90  SMW  ISMD 

F.E.  Warren 


DOTI 
DO  2  2 
DO  9 


91  SMW  ISMD 

Minot  DOTI 

DO  2  2 
DO  9 


308  SMW  ISMD 

Little  Rock 


DOTI 


Big  concern  within  SAC  about 
reassessing  CRI  -  must  give 
MCCMs  greater  depth  of  knowledge. 
There  is  flexibility  and 
autonomy  with  ISD 

EWO  training  attempts  to  use  ISD 
in  a  modified  manner,  however,  we 
teach  all  required  subjects  not 
because  crew  force  needs  that 
much  training,  but  because 
SAC  regulations,  the  importance 
of  minimal  errors  in  EWO,  and  the 
never  ending  requirement  to  impress 
inspectors  causes  each  EWO  Training 
Office  to  train  beyond  standards 
and  requirements 


Only  real  benefit  of  ISD  at  unit  is 
for  courseware  developers  -  it  is 
a  roadmap  for  lesson  planning 


Not  enough  strife  to  train  above  CPOs 


ISD  forced  on  units,  don't  have  any 
real  way  of  comparing  with  something 
else 

ISD  theory  is  good  information,  but 
not  real  useful  or  effective  to 
practice  at  missile  unit.  Students 
must  be  trained  as  crews,  not 
ind ividuals 


DO  2  2 


Unit 


Respondent 


Comments 


321  SMW 

ISMD 

Missiles  train  minimum,  but  nowhere 

Grand  Porks 

does  ISD  say  anything  about  minimum. 
(SAC  perhaps  has  misinterpreted 
intention  of  ISD.) 

DOTI 

— 

DO  22 

HQ  SAC  doesn't  understand  and  isn't 
concerned  with  ISD  process  - 
Inspectors  allow  no  ISD-related 
innovation 

DO  9 

—  — 

341  SMW 

ISMD 

— 

Malms trom 

DOTI 

ISD  good  in  perfect  world  of  ATC  - 
harder  to  apply  at  operational  unit 

D022 

— 

DO  9 

— 

351  SMW 

ISMD 

ISD  is  just  a  tool,  not  a  panacea; 

Whiteman 

Must  make  it  clear  that  individual 
is  responsible  for  own  proficiency 

DOTI 

— 

DO  2  2 

— 

DO  9 

Missile  units  don't  really  have 
flexibility  offered  by  ISD.  Fast 
students  and  slow  students  still 
must  progress  at  same  pace. 

381  SMW 

ISMD 

Avid  believer  in  ISD.  Big  problem 

McConnell 

is  evaluator  acceptance  of  it  - 
everyone  must  commit  to  it 

DOTI 

Good  in  theory,  not  totally 
applicable  -  doesn' t  really  help 
instill  system  knowledge.  Self¬ 
pacing  also  hard  to  do  -  can' t  depen 
on  students  in  a  weapon 
system  of  critical  nature. 

DO  2  2 
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